arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2208.06458cs.CLcs.LG
收藏

LM-CORE:具有上下文相关外部知识的语言模型

LM-CORE: Language Models with Contextually Relevant External Knowledge

Microsoft Research(微软研究院);Adobe(奥多比);Delhi Technological University(德里理工大学)

Jivat Neet Kaur, Sumit Bhatia, Milan Aggarwal, Rachit Bansal, Balaji Krishnamurthy

摘要:

基于transformer的大型预训练语言模型在各种知识密集型任务上取得了令人印象深刻的性能,并且可以在其参数中捕获事实知识。我们认为,在模型参数中存储大量的知识是次优的知识和资源需求量不断增长。我们认为,一个更有效的替代方案是提供显式访问上下文相关的结构化知识的模型和训练它使用这些知识。我们提出了LM-CORE --一个实现这一目标的通用框架--它允许从外部知识源训练语言模型的\textit{decoupling},并允许后者在不影响已经训练好的模型的情况下进行更新。实验结果表明,LM-CORE,具有访问外部知识,实现了显着的和强大的性能优于国家的最先进的知识增强语言模型的知识探测任务,可以有效地处理知识更新,并表现良好的两个下游任务。我们还提出了一个彻底的错误分析,突出了LM-CORE的成功和失败。

英文摘要:

Large transformer-based pre-trained language models have achieved impressive performance on a variety of knowledge-intensive tasks and can capture factual knowledge in their parameters. We argue that storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of knowledge and resource requirements. We posit that a more efficient alternative is to provide explicit access to contextually relevant structured knowledge to the model and train it to use that knowledge. We present LM-CORE -- a general framework to achieve this -- that allows \textit{decoupling} of the language model training from the external knowledge source and allows the latter to be updated without affecting the already trained model. Experimental results show that LM-CORE, having access to external knowledge, achieves significant and robust outperformance over state-of-the-art knowledge-enhanced language models on knowledge probing tasks; can effectively handle knowledge updates; and performs well on two downstream tasks. We also present a thorough error analysis highlighting the successes and failures of LM-CORE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑