arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.00083cs.CLcs.IR
收藏

上下文检索增强语言模型

In-Context Retrieval-Augmented Language Models

AI21 Labs(AI21实验室)

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham

AI总结:

本文提出In-Context RALM,无需修改架构或训练,直接前置检索文档至输入,显著提升多规模LM性能,并支持特化检索以进一步优化。

摘要:

检索增强语言建模(RALM)方法在生成期间利用基础语料库中的相关文档对语言模型(LM)进行条件化,已被证明能显著提升语言建模性能。此外,它们能缓解事实性不准确文本生成的问题,并提供自然的来源归因机制。现有的RALM方法侧重于修改LM架构以便于融入外部信息,这显著增加了部署的复杂性。本文考虑了一种简单的替代方案,我们称之为In-Context RALM:保持LM架构不变,将基础文档前置到输入中,无需对LM进行任何进一步训练。我们展示了基于现成通用检索器的In-Context RALM在各种模型规模和多样化语料库上提供了惊人的巨大LM增益。我们还证明了文档检索和排序机制可以专门针对RALM设置进行特化,以进一步提升性能。我们得出结论,In-Context RALM在提高LM基础化(grounding)的普及程度方面具有巨大潜力,特别是在必须不加修改地使用预训练LM、甚至通过API访问的场景中。

英文摘要:

Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM. We show that In-Context RALM that builds on off-the-shelf general purpose retrievers provides surprisingly large LM gains across model sizes and diverse corpora. We also demonstrate that the document retrieval and ranking mechanism can be specialized to the RALM setting to further boost performance. We conclude that In-Context RALM has considerable potential to increase the prevalence of LM grounding, particularly in settings where a pretrained LM must be used without modification or even via API access.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑