arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.03187cs.CLcs.AIcs.LG
收藏

具备可靠性、适应性与可归因性的检索增强语言模型

Reliable, Adaptable, and Attributable Language Models with Retrieval

University of Washington(华盛顿大学);Meta AI;Princeton University(普林斯顿大学);Allen Institute for AI(艾伦人工智能研究所)

Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, Wen-tau Yih

AI总结:

该立场论文针对参数化语言模型的幻觉、适配难、不可验证等问题,提出以检索增强语言模型作为下一代方案,分析其应用障碍并给出涵盖组件优化、交互改进、基建投入的通用发展路线图。

摘要:

基于海量网络数据训练的参数化语言模型(LMs)展现出出色的灵活性与能力,但仍存在幻觉、难以适配新数据分布、缺乏可验证性等实际挑战。在这篇立场论文中,我们主张用检索增强语言模型取代参数化语言模型,作为下一代语言模型。通过在推理阶段引入大规模数据存储,检索增强语言模型可具备更强的可靠性、适应性与可归因性。尽管潜力显著,检索增强语言模型仍因诸多障碍尚未得到广泛应用:具体而言,当前的检索增强语言模型难以在问答等知识密集型任务之外利用有效文本,检索组件与语言模型组件的交互有限,且缺乏规模化所需的基础设施。为解决这些问题,我们提出了通用检索增强语言模型的发展路线图,包括重新审视数据存储与检索器、探索检索器与语言模型交互更优的流水线,以及大力投入高效训练与推理的基础设施建设。

英文摘要:

Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑