arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.15915cs.CLcs.IR
收藏

参数化检索增强生成

Parametric Retrieval Augmented Generation

Tsinghua University(清华大学)

Weihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang, Hongning Wang, Ziyi Ye, Yujia Zhou, Yiqun Liu

AI总结:

本文提出参数化检索增强生成(Parametric RAG)新范式,通过文档参数化将外部知识直接整合到LLM前馈网络参数中,克服了传统上下文注入方法的局限,显著提升了知识增强的效率与效果,且可与现有方法结合。

摘要:

检索增强生成(RAG)技术通过解决幻觉、知识过时和领域适应等问题,成为增强大型语言模型(LLMs)可靠性的有前途的方案。具体而言,现有的RAG方法将从外部语料库或数据库检索到的相关文档附加到LLMs的输入中,以指导其生成过程,我们将此称为上下文知识注入方法。尽管该方法简单且通常有效,但存在固有的局限性。首先,增加上下文长度和相关文档数量会导致更高的计算开销和性能下降,尤其是在复杂推理任务中。更重要的是,上下文知识注入主要在输入层面操作,而LLMs将内部知识存储在其参数中。这种差距从根本上限制了上下文方法的能力。为此,我们提出了参数化检索增强生成(Parametric RAG),这是一种新的RAG范式,通过文档参数化将外部知识直接整合到LLM的前馈网络(FFN)参数中。该方法不仅通过消除将多个文档注入LLMs输入上下文的需要而节省了在线计算成本,还深化了外部知识向LLM参数化知识空间的整合。实验结果表明,Parametric RAG显著提升了LLMs知识增强的有效性和效率。此外,它可以与上下文RAG方法结合以实现更佳的性能。我们已在以下匿名GitHub链接开源了所有代码、数据和模型:https://github.com/oneal2000/PRAG

英文摘要:

Retrieval-augmented generation (RAG) techniques have emerged as a promising solution to enhance the reliability of large language models (LLMs) by addressing issues like hallucinations, outdated knowledge, and domain adaptation. In particular, existing RAG methods append relevant documents retrieved from external corpus or databases to the input of LLMs to guide their generation process, which we refer to as the in-context knowledge injection method. While this approach is simple and often effective, it has inherent limitations. Firstly, increasing the context length and number of relevant documents can lead to higher computational overhead and degraded performance, especially in complex reasoning tasks. More importantly, in-context knowledge injection operates primarily at the input level, but LLMs store their internal knowledge in their parameters. This gap fundamentally limits the capacity of in-context methods. To this end, we introduce Parametric retrieval-augmented generation (Parametric RAG), a new RAG paradigm that integrates external knowledge directly into the parameters of feed-forward networks (FFN) of an LLM through document parameterization. This approach not only saves online computational costs by eliminating the need to inject multiple documents into the LLMs' input context, but also deepens the integration of external knowledge into the parametric knowledge space of the LLM. Experimental results demonstrate that Parametric RAG substantially enhances both the effectiveness and efficiency of knowledge augmentation in LLMs. Also, it can be combined with in-context RAG methods to achieve even better performance. We have open-sourced all the code, data, and models in the following anonymized GitHub link: https://github.com/oneal2000/PRAG

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑