Link Prediction (LP) approaches based on Language Models (LMs) operate over the labels and descriptions of entities and relations in a Knowledge Graph (KG). Recent approaches have shown that incorporating a local graph neighborhood can improve the LP capabilities of LMs. These approaches usually sample a context from the neighborhood around a query triple randomly, thereby incorporating noise that might hinder the model in making correct predictions.

While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched.Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs -which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structurebut GNNs cannot represent text features as well as pretrained LMs.In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses.The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets.Simultaneously, we design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph.This enables GLMs to process graphs, texts, and interleaved inputs of both.Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM-and GNN-based baselines in supervised and zeroshot setting, demonstrating their versatility.1

Arguments often do not make explicit how a conclusion follows from its premises.To compensate for this lack, we enrich arguments with structured background knowledge to support knowledge-intense argumentation tasks.We present a new unsupervised method for constructing Contextualized Commonsense Knowledge Graphs (CCKGs) that selects contextually relevant knowledge from large knowledge graphs (KGs) efficiently and at high quality.Our work goes beyond context-insensitive knowledge extraction heuristics by computing semantic similarity between KG triplets and textual arguments.Using these triplet similarities as weights, we extract contextualized knowledge paths that connect a conclusion to its premise, while maximizing similarity to the argument.We combine multiple paths into a CCKG that we optionally prune to reduce noise and raise precision.Intrinsic evaluation of the quality of our graphs shows that our method is effective for (re)constructing human explanation graphs.Manual evaluations in a large-scale knowledge selection setup confirm high recall and precision of implicit CSK in the CCKGs.Finally, we demonstrate the effectiveness of CCKGs in a knowledge-insensitive argument quality rating task, outperforming strong baselines and rivaling a GPT-3 based system. 1