LIT-RAGBench：大型语言模型检索增强生成能力的基准测试
LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
AI总结：
LIT-RAGBench通过五个类别评估生成器的整合、推理、逻辑、表格和回避能力，提供统一的基准测试框架，用于评估多方面能力并指导模型选择与改进。
摘要：
检索增强生成（RAG）是一种框架，其中生成器（如大型语言模型）通过检索外部文档集合来生成答案。在实践中，生成器必须整合长上下文信息、进行多步骤推理、解读表格并在证据缺失时回避。然而，现有生成器基准测试的覆盖范围有限，无法同时评估多种能力。为弥合现有评估与实际应用之间的差距，我们引入LIT-RAGBench（逻辑、整合、表格、推理和回避RAG生成器基准），定义了五个类别：整合、推理、逻辑、表格和回避，每个类别进一步分为实际评估方面。LIT-RAGBench系统地覆盖了跨类别的多方面模式组合。通过使用虚构实体和场景，LIT-RAGBench评估基于提供的外部文档的答案。数据集包含114个由人类构建的日本问题及通过机器翻译生成的英文版本。我们使用LLM作为评判者进行评分，并报告类别和整体准确率。在API基于和开放权重模型中，没有模型超过90%的整体准确率。通过在每个类别中衡量优势和劣势，LIT-RAGBench成为实际RAG部署中模型选择和构建专用RAG模型的重要指标。我们在此发布LIT-RAGBench，包括数据集和评估代码，网址为https://github.com/Koki-Itai/LIT-RAGBench。
英文摘要：
Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving documents from an external collection using a Retriever. In practice, Generators must integrate evidence from long contexts, perform multi-step reasoning, interpret tables, and abstain when evidence is missing. However, existing benchmarks for Generators provide limited coverage, with none enabling simultaneous evaluation of multiple capabilities under unified conditions. To bridge the gap between existing evaluations and practical use, we introduce LIT-RAGBench (the Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), which defines five categories: Integration, Reasoning, Logic, Table, and Abstention, each further divided into practical evaluation aspects. LIT-RAGBench systematically covers patterns combining multiple aspects across categories. By using fictional entities and scenarios, LIT-RAGBench evaluates answers grounded in the provided external documents. The dataset consists of 114 human-constructed Japanese questions and an English version generated by machine translation with human curation. We use LLM-as-a-Judge for scoring and report category-wise and overall accuracy. Across API-based and open-weight models, no model exceeds 90% overall accuracy. By making strengths and weaknesses measurable within each category, LIT-RAGBench serves as a valuable metric for model selection in practical RAG deployments and for building RAG-specialized models. We release LIT-RAGBench, including the dataset and evaluation code, at https://github.com/Koki-Itai/LIT-RAGBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
