arXiv 2503.003532025-03-01cs.CLcs.IR

U-NIAH：面向长上下文大海捞针的统一 RAG 与 LLM 评估

U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack

Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；College of Design and Innovation, Tongji University（同济大学设计与创新学院）

Yunfan Gao, Yun Xiong, Wenlong Wu, Zijing Huang, Bohan Li, Haofen Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出统一框架 U-NIAH，在受控长上下文设置下系统比较 LLM 与 RAG，发现 RAG 显著提升小型 LLM 鲁棒性并取得 82.58% 胜率，但揭示了检索噪声、反向块排序及高级推理模型兼容性下降等问题。

摘要：

大型语言模型（LLMs）的最新进展将其上下文窗口扩展到前所未有的长度，引发了关于检索增强生成（RAG）必要性的争论。为了解决现有大海捞针（NIAH）评估范式碎片化和测试用例有限的问题，本文提出了 U-NIAH，这是一个在受控长上下文设置下系统比较 LLM 和 RAG 方法的统一框架。我们的框架超越了传统的 NIAH，纳入了多针、长针和针中针配置以及不同的检索设置，同时利用合成的 Starlight Academy 数据集（一个虚构的魔法宇宙）来消除预训练知识带来的偏差。通过大量实验，我们研究了三个研究问题：(1) LLM 与 RAG 之间的性能权衡，(2) RAG 中的错误模式，以及 (3) RAG 在复杂设置中的局限性。研究结果表明，RAG 通过缓解“中间迷失”效应并提高鲁棒性，显著增强了较小的 LLM，对 LLM 取得了 82.58% 的胜率。然而，我们观察到检索噪声和反向块排序会降低性能，而令人惊讶的是，高级推理 LLM 由于对语义干扰项敏感，表现出 RAG 兼容性下降。我们识别了典型的错误模式，包括因噪声导致的遗漏、高噪声临界条件下的幻觉以及自我怀疑行为。我们的工作不仅突出了 RAG 和 LLM 的互补作用，还为优化部署提供了可操作的见解。代码：https://github.com/Tongji-KGLLM/U-NIAH。

英文摘要：

Recent advancements in Large Language Models (LLMs) have expanded their context windows to unprecedented lengths, sparking debates about the necessity of Retrieval-Augmented Generation (RAG). To address the fragmented evaluation paradigms and limited cases in existing Needle-in-a-Haystack (NIAH), this paper introduces U-NIAH, a unified framework that systematically compares LLMs and RAG methods in controlled long context settings. Our framework extends beyond traditional NIAH by incorporating multi-needle, long-needle, and needle-in-needle configurations, along with different retrieval settings, while leveraging the synthetic Starlight Academy dataset-a fictional magical universe-to eliminate biases from pre-trained knowledge. Through extensive experiments, we investigate three research questions: (1) performance trade-offs between LLMs and RAG, (2) error patterns in RAG, and (3) RAG's limitations in complex settings. Our findings show that RAG significantly enhances smaller LLMs by mitigating the "lost-in-the-middle" effect and improving robustness, achieving an 82.58% win-rate over LLMs. However, we observe that retrieval noise and reverse chunk ordering degrade performance, while surprisingly, advanced reasoning LLMs exhibit reduced RAG compatibility due to sensitivity to semantic distractors. We identify typical error patterns including omission due to noise, hallucination under high noise critical condition, and self-doubt behaviors. Our work not only highlights the complementary roles of RAG and LLMs, but also provides actionable insights for optimizing deployments. Code: https://github.com/Tongji-KGLLM/U-NIAH.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。