arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.09306cs.CL
收藏

通过蒸馏将知识更新传播至语言模型

Propagating Knowledge Updates to LMs Through Distillation

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Shankar Padmanabhan, Yasumasa Onoe, Michael J. Q. Zhang, Greg Durrett, Eunsol Choi

AI总结:

本文提出一种基于上下文蒸馏的两阶段方法,通过生成迁移集并匹配学生与教师模型分布来更新语言模型知识,实验证明其知识传播效果优于微调和其他梯度编辑方法,且注入多达150个实体定义也不损害其他上下文性能。

摘要:

现代语言模型具备存储和使用关于现实世界实体的大量知识的能力,但如何更新存储在模型参数中的此类知识仍不清楚。先前更新语言模型中知识的方法虽然能成功注入原子事实,但更新后的语言模型无法基于注入的事实进行推理。在本工作中,我们证明了一种基于上下文蒸馏的方法既能传授关于实体的知识,又能传播该知识以支持更广泛的推理。我们的方法包含两个阶段:迁移集生成和在迁移集上进行蒸馏。我们首先通过提示语言模型根据实体定义生成续写内容来生成迁移集。然后,我们更新模型参数,使语言模型(学生)在迁移集上的分布与以定义为条件的语言模型(教师)的分布相匹配。我们的实验表明,与微调及其他基于梯度的知识编辑方法相比,该方法在传播知识更新方面更为有效。此外,即使在一次性注入多达150个实体的定义时,它也不会损害其他上下文中的性能。

英文摘要:

Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored in model parameters. While prior methods for updating knowledge in LMs successfully inject atomic facts, updated LMs fail to make inferences based on injected facts. In this work, we demonstrate that a context distillation-based approach can both impart knowledge about entities and propagate that knowledge to enable broader inferences. Our approach consists of two stages: transfer set generation and distillation on the transfer set. We first generate a transfer set by prompting a language model to generate continuations from the entity definition. Then, we update the model parameters so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than fine-tuning and other gradient-based knowledge-editing methods. Moreover, it does not compromise performance in other contexts, even when injecting the definitions of up to 150 entities at once.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑