arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.26356cs.CL
收藏

检索增强生成的上下文优化:梯度下降视角

In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective

University of Massachusetts, Amherst(马萨诸塞大学阿默斯特分校);University of Connecticut(康涅狄格大学);Emory University(埃默里大学)

Mingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao, Hong Yu

AI总结:

本文从梯度下降视角研究检索增强生成(RAG)作为上下文优化过程,提出一种轻量级前向更新方法,在冻结LLM和检索器的情况下提升生成器对检索证据的利用。

摘要:

上下文学习最近被与线性自注意力模型中的隐式梯度下降联系起来,表明上下文可以诱导前向传递更新。检索增强生成(RAG)也依赖于上下文,但检索到的文档通常被视为静态证据而非适应信号。我们将RAG研究为一种上下文优化过程。首先,我们展示一个线性自注意力层可以在统一的线性化RAG目标上实现一步梯度下降,该目标涵盖基于投影和基于点积的检索接口。这给出了检索增强预测与上下文优化一致的一个精确区域。我们使用这一结果并非作为LLM计算的字面模型,而是作为调整查询与检索证据之间交互的指南。然后,我们测试这种对应关系的边界:在受控的线性扩展下保持稳定,但在非线性架构下变得依赖于特征分布。最后,我们将这一观点转化为一种针对冻结RAG LLM的轻量级方法。该方法保持检索器和骨干网络固定,并预测一个上下文条件更新到生成器侧的证据使用接口。在七个QA基准、两个检索器和两个冻结LLM骨干网络上,这种仅前向的更新改进了共享接口基线,迁移到未见任务,并以更低的每查询成本接近测试时的梯度适应。

英文摘要:

In-context learning has recently been linked to implicit gradient descent in linear self-attention models, suggesting that context can induce a forward-pass update. Retrieval-augmented generation (RAG) also relies on context, but retrieved documents are usually treated as static evidence rather than signals for adaptation. We study RAG as an in-context optimization process. First, we show that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective covering both projection-based and dot-product retrieval interfaces. This gives an exact regime where retrieval-augmented prediction and in-context optimization coincide. We use this result not as a literal model of LLM computation, but as a guide for adapting the interaction between queries and retrieved evidence. We then test the boundary of this correspondence: it remains stable under controlled linear extensions, but becomes feature-distribution dependent under nonlinear architectures. Finally, we turn this view into a lightweight method for frozen RAG LLMs. The method keeps the retriever and backbone fixed, and predicts a context-conditioned update to a generator-side evidence-use interface. Across seven QA benchmarks, two retrievers, and two frozen LLM backbones, this forward-only update improves a shared-interface baseline, transfers to held-out tasks, and approaches test-time gradient adaptation at much lower per-query cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑