LaRA：检索增强生成与长上下文LLM的基准测试——LC或RAG路由没有银弹
LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing
AI总结：
提出LaRA基准，通过2326个测试用例系统比较RAG与LC LLMs，发现两者最优选择取决于模型参数、长文本能力、上下文长度、任务类型及检索片段特征等复杂因素，为LLM应用部署提供实用指南。
摘要：
将外部知识有效融入大语言模型（LLMs）对于增强其能力并满足现实需求至关重要。检索增强生成（RAG）通过检索最相关的片段输入LLMs，为实现这一目标提供了有效方法。然而，LLMs上下文窗口大小的进展提供了另一种替代方案，引发了RAG在有效处理外部知识时是否仍然必要的疑问。现有几项研究对RAG与长上下文（LC）LLMs的比较尚无定论，主要归因于基准设计的局限性。本文提出LaRA，一个专为严格比较RAG和LC LLMs设计的新颖基准。LaRA涵盖2326个测试用例，横跨四个实际QA任务类别和三种自然产生的长文本类型。通过对七个开源和四个专有LLMs的系统评估，我们发现RAG与LC之间的最佳选择取决于多种因素的复杂相互作用，包括模型参数大小、长文本能力、上下文长度、任务类型以及检索片段的特征。我们的发现为从业者在开发和部署LLM应用时有效利用RAG和LC方法提供了可操作的指南。代码和数据集见：https://github.com/Alibaba-NLP/LaRA。
英文摘要：
Effectively incorporating external knowledge into Large Language Models (LLMs) is crucial for enhancing their capabilities and addressing real-world needs. Retrieval-Augmented Generation (RAG) offers an effective method for achieving this by retrieving the most relevant fragments into LLMs. However, the advancements in context window size for LLMs offer an alternative approach, raising the question of whether RAG remains necessary for effectively handling external knowledge. Several existing studies provide inconclusive comparisons between RAG and long-context (LC) LLMs, largely due to limitations in the benchmark designs. In this paper, we present LaRA, a novel benchmark specifically designed to rigorously compare RAG and LC LLMs. LaRA encompasses 2326 test cases across four practical QA task categories and three types of naturally occurring long texts. Through systematic evaluation of seven open-source and four proprietary LLMs, we find that the optimal choice between RAG and LC depends on a complex interplay of factors, including the model's parameter size, long-text capabilities, context length, task type, and the characteristics of the retrieved chunks. Our findings provide actionable guidelines for practitioners to effectively leverage both RAG and LC approaches in developing and deploying LLM applications. Our code and dataset is provided at: \href{https://github.com/Alibaba-NLP/LaRA}{\textbf{https://github.com/Alibaba-NLP/LaRA}}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
