arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.11564cs.CLcs.AI
收藏

将知识从参数中解耦以实现即插即用的语言建模

Decouple knowledge from parameters for plug-and-play language modeling

Peking University(北京大学);Renmin University of China(中国人民大学);BIGAI(北京通用人工智能研究院)

Xin Cheng, Yankai Lin, Xiuying Chen, Dongyan Zhao, Rui Yan

AI总结:

本文提出PlugLM,通过可微即插即用记忆将知识存储与模型参数解耦,实现可编辑、可扩展且可解释的知识利用,并在领域适应、知识更新和任务内知识学习三个场景中验证了其有效性。

摘要:

预训练语言模型(PLM)在各种自然语言处理任务中取得了令人瞩目的成果。已有研究揭示,其成功的关键因素之一是这些模型的参数在预训练过程中隐式地学习了各种知识。然而,在模型参数中隐式编码知识存在两个根本性缺陷。首先,一旦模型训练完成,知识既不可编辑也不可扩展,这在知识持续演进的背景下尤其成问题。其次,这种方式缺乏可解释性,使人类无法理解PLM在解决特定问题时需要哪些知识。在本文中,我们提出了PlugLM,一种具有可微分即插即用记忆(DPM)的预训练模型。其核心思想是通过可编辑、可扩展的键值记忆将知识存储与模型参数解耦,并通过在DPM中进行知识检索以可解释的方式利用知识。为了验证这一设计选择,我们在三种设置下进行了评估,包括:(1)领域适应。PlugLM在四个领域上平均获得3.95的F1提升,且无需任何领域内预训练。(2)知识更新。PlugLM可以在预训练完成后以无需训练的方式吸收新知识。(3)任务内知识学习。PlugLM可以通过知识提示将训练样本纳入DPM来进一步提升性能。

英文摘要:

Pre-trained language models(PLM) have made impressive results in various NLP tasks. It has been revealed that one of the key factors to their success is the parameters of these models implicitly learn all kinds of knowledge during pre-training. However, encoding knowledge implicitly in the model parameters has two fundamental drawbacks. First, the knowledge is neither editable nor scalable once the model is trained, which is especially problematic in that knowledge is consistently evolving. Second, it lacks interpretability and prevents humans from understanding which knowledge PLM requires for a certain problem. In this paper, we introduce PlugLM, a pre-training model with differentiable plug-in memory(DPM). The key intuition is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the DPM. To justify this design choice, we conduct evaluations in three settings including: (1) domain adaptation. PlugLM obtains 3.95 F1 improvements across four domains on average without any in-domain pre-training. (2) knowledge update. PlugLM could absorb new knowledge in a training-free way after pre-training is done. (3) in-task knowledge learning. PlugLM could be further improved by incorporating training samples into DPM with knowledge prompting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑