arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.09170cs.CLcs.AI
收藏

KILM:知识注入到编码器-解码器语言模型

KILM: Knowledge Injection into Encoder-Decoder Language Models

Amazon Alexa AI(亚马逊Alexa人工智能);Hong Kong University of Science and Technology(香港科技大学)

Yan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar, Yang Liu, Dilek Hakkani-Tür

摘要:

大型预训练语言模型(PLM)已被证明在其参数内保留隐式知识。为了增强这种隐含的知识,我们提出了知识注入到语言模型(KILM),一种新的方法,注入实体相关的知识到编码器-解码器PLM,通过生成知识填充目标,通过持续的预训练。这是在不对PLM进行架构修改或添加额外参数的情况下完成的。在一套跨越众多数据集的知识密集型任务上的实验结果表明,KILM使模型能够保留更多的知识,减少幻觉,同时保持其在一般NLU和NLG任务上的原始性能。KILM还展示了在实体消歧等任务上改进的zero-shot性能,优于具有30倍以上参数的最先进模型。

英文摘要:

Large pre-trained language models (PLMs) have been shown to retain implicit knowledge within their parameters. To enhance this implicit knowledge, we propose Knowledge Injection into Language Models (KILM), a novel approach that injects entity-related knowledge into encoder-decoder PLMs, via a generative knowledge infilling objective through continued pre-training. This is done without architectural modifications to the PLMs or adding additional parameters. Experimental results over a suite of knowledge-intensive tasks spanning numerous datasets show that KILM enables models to retain more knowledge and hallucinate less, while preserving their original performance on general NLU and NLG tasks. KILM also demonstrates improved zero-shot performances on tasks such as entity disambiguation, outperforming state-of-the-art models having 30x more parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑