2019/06/17 by Robert L. Logan, Nelson F. Liu, Logan, Robert L. +7 · 1 citation
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Advanced Graph Neural Networks
paper · pdf · doi:10.48550/arxiv.1906.07241
Modeling human language requires the ability to not only generate fluent text\nbut also encode factual knowledge. However, traditional language models are\nonly capable of remembering facts seen at training time, and often have\ndifficulty recalling them. To address this, we introduce the knowledge graph\nlanguage model (KGLM), a neural language model with mechanisms for selecting\nand copying facts from a knowledge graph that are relevant to the context.\nThese mechanisms enable the model to render information it has never seen\nbefore, as well as generate out-of-vocabulary tokens. We also introduce the\nLinked WikiText-2 dataset, a corpus of annotated text aligned to the Wikidata\nknowledge graph whose contents (roughly) match the popular WikiText-2\nbenchmark. In experiments, we demonstrate that the KGLM achieves significantly\nbetter performance than a strong baseline language model. We additionally\ncompare different language model's ability to complete sentences requiring\nfactual knowledge, showing that the KGLM outperforms even very large language\nmodels in generating facts.\n