arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.04726cs.CLcs.AIcs.LG
收藏

知识库:通过软库向语言模型注入世界知识

Knowledge Prompts: Injecting World Knowledge into Language Models through Soft Prompts

Google Research(谷歌研究院)

Cicero Nogueira dos Santos, Zhe Dong, Daniel Cer, John Nham, Siamak Shakeri, Jianmo Ni, Yun-hsuan Sung

摘要:

软提示最近已被提出作为一种工具,以适应大型冻结语言模型(LM)的新任务。在这项工作中,我们将软提示重新用于将世界知识注入LM的任务。我们介绍了一种方法,训练软提示,通过自监督学习的数据从知识库。由此产生的软知识提示(KPs)是独立于任务的,并作为一个外部存储器的LM。我们进行定性和定量的实验,并证明:(1)KPs可以有效地建模的训练数据的结构;(2)KPs可以用来提高LM在不同的知识密集型任务的性能。

英文摘要:

Soft prompts have been recently proposed as a tool for adapting large frozen language models (LMs) to new tasks. In this work, we repurpose soft prompts to the task of injecting world knowledge into LMs. We introduce a method to train soft prompts via self-supervised learning on data from knowledge bases. The resulting soft knowledge prompts (KPs) are task independent and work as an external memory of the LMs. We perform qualitative and quantitative experiments and demonstrate that: (1) KPs can effectively model the structure of the training data; (2) KPs can be used to improve the performance of LMs in different knowledge intensive tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑