在长上下文语言模型时代为RAG辩护
In Defense of RAG in the Era of Long-Context Language Models
AI总结：
本文针对长上下文LLM中相关信息关注度下降的问题，提出保序检索增强生成机制OP-RAG，通过倒U形曲线上的最佳点以更少token实现更优答案质量。
摘要：
为了克服早期大语言模型（LLM）有限的上下文限制，检索增强生成（RAG）在过去一直是基于上下文的答案生成的一种可靠解决方案。最近，长上下文LLM的出现使模型能够纳入更长的文本序列，从而使RAG的吸引力下降。近期研究表明，长上下文LLM在长上下文应用中显著优于RAG。与现有偏向长上下文LLM而非RAG的研究不同，我们认为LLM中极长的上下文会导致对相关信息的关注度减弱，并可能造成答案质量的下降。本文重新审视了长上下文答案生成中的RAG。我们提出了一种保序检索增强生成（OP-RAG）机制，该机制显著提升了RAG在长上下文问答应用中的性能。使用OP-RAG时，随着检索到的文本块数量增加，答案质量先上升后下降，形成一条倒U形曲线。存在一些最佳点，在这些点上OP-RAG能够以比将整个上下文作为输入的长上下文LLM少得多的token实现更高的答案质量。在公开基准上的大量实验证明了我们的OP-RAG的优越性。
英文摘要：
Overcoming the limited context limitations in early-generation LLMs, retrieval-augmented generation (RAG) has been a reliable solution for context-based answer generation in the past. Recently, the emergence of long-context LLMs allows the models to incorporate much longer text sequences, making RAG less attractive. Recent studies show that long-context LLMs significantly outperform RAG in long-context applications. Unlike the existing works favoring the long-context LLM over RAG, we argue that the extremely long context in LLMs suffers from a diminished focus on relevant information and leads to potential degradation in answer quality. This paper revisits the RAG in long-context answer generation. We propose an order-preserve retrieval-augmented generation (OP-RAG) mechanism, which significantly improves the performance of RAG for long-context question-answer applications. With OP-RAG, as the number of retrieved chunks increases, the answer quality initially rises, and then declines, forming an inverted U-shaped curve. There exist sweet points where OP-RAG could achieve higher answer quality with much less tokens than long-context LLM taking the whole context as input. Extensive experiments on public benchmark demonstrate the superiority of our OP-RAG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
