巴拉克的妻子希拉里:使用知识图谱进行事实感知语言建模
Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
摘要:
对人类语言进行建模不仅需要生成流畅的文本,还需要对事实知识进行编码。然而,传统的语言模型只能记住在训练时看到的事实,并且通常难以回忆它们。为了解决这个问题,我们引入了知识图谱语言模型(KGLM),这是一种神经语言模型,具有从与上下文相关的知识图谱中选择和复制事实的机制。这些机制使模型能够呈现以前从未见过的信息,并生成词汇表外的标记。我们还介绍了Linked WikiText-2数据集,这是一个与Wikidata知识图谱对齐的注释文本语料库,其内容(大致)与流行的WikiText-2基准相匹配。在实验中,我们证明了KGLM实现了显着更好的性能比一个强大的基线语言模型。我们还比较了不同的语言模型完成需要事实知识的句子的能力,表明KGLM在生成事实方面甚至优于非常大的大语言模型。
英文摘要:
Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge. However, traditional language models are only capable of remembering facts seen at training time, and often have difficulty recalling them. To address this, we introduce the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. We also introduce the Linked WikiText-2 dataset, a corpus of annotated text aligned to the Wikidata knowledge graph whose contents (roughly) match the popular WikiText-2 benchmark. In experiments, we demonstrate that the KGLM achieves significantly better performance than a strong baseline language model. We additionally compare different language model's ability to complete sentences requiring factual knowledge, showing that the KGLM outperforms even very large language models in generating facts.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。