RAGCap-Bench: 评估代理检索增强生成系统中LLM能力的基准测试
RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems
AI总结：
本文提出RAGCap-Bench，用于评估代理检索增强生成系统中中间任务的细粒度能力，通过分析现有系统输出识别常见任务和核心能力，设计针对性评估问题，实验表明增强中间能力的模型能获得更好的整体性能。
摘要：
检索增强生成（RAG）通过动态检索外部信息缓解大型语言模型（LLMs）的关键限制，如事实错误、过时知识和幻觉。最近的研究通过代理RAG系统扩展了这一范式，其中LLMs作为代理迭代地计划、检索和推理复杂查询。然而，这些系统在处理具有挑战性的多跳问题时仍存在困难，且其中间推理能力仍缺乏深入研究。为此，我们提出了RAGCap-Bench，一个以能力为导向的基准测试，用于对代理RAG工作流程中的中间任务进行细粒度评估。我们分析了最先进系统的输出，以识别常见任务和执行所需的核心能力，然后构建了一个典型LLM错误的分类学，以设计针对性的评估问题。实验表明，具有更强RAGCap性能的“慢思考”模型在端到端结果上表现更好，这证明了该基准测试的有效性以及增强这些中间能力的重要性。
英文摘要：
Retrieval-Augmented Generation (RAG) mitigates key limitations of Large Language Models (LLMs)-such as factual errors, outdated knowledge, and hallucinations-by dynamically retrieving external information. Recent work extends this paradigm through agentic RAG systems, where LLMs act as agents to iteratively plan, retrieve, and reason over complex queries. However, these systems still struggle with challenging multi-hop questions, and their intermediate reasoning capabilities remain underexplored. To address this, we propose RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs from state-of-the-art systems to identify common tasks and the core capabilities required for their execution, then construct a taxonomy of typical LLM errors to design targeted evaluation questions. Experiments show that "slow-thinking" models with stronger RAGCap performance achieve better end-to-end results, underscoring the benchmark's validity and the importance of enhancing these intermediate capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
