CoFE-RAG：一种具有增强数据多样性的检索增强生成全链路综合评估框架
CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
AI总结：
针对RAG系统评估中数据多样性有限、问题定位困难和检索评估不稳定的问题，提出全链路综合评估框架CoFE-RAG，引入多粒度关键词评估检索上下文并发布多样化基准数据集，为系统各阶段提供细致的评估与见解。
摘要：
检索增强生成（RAG）旨在借助从外部知识源检索到的上下文来增强大型语言模型（LLM），从而生成更准确、更可靠的答案，进而降低幻觉的发生率。尽管取得了诸多进展，但由于以下问题，评估这些系统仍是一个关键的研究领域：（1）数据多样性有限：知识源和查询类型的多样性不足限制了RAG系统的适用性；（2）问题定位困难：现有的评估方法难以准确定位RAG流程中出现问题的阶段；（3）检索评估不稳定：当分块策略发生变化时，这些方法往往无法有效评估检索性能。为了应对这些挑战，我们提出了一种全链路综合评估（CoFE-RAG）框架，以促进对整个RAG流程（包括分块、检索、重排序和生成）的全面评估。为了有效评估前三个阶段，我们引入了多粒度关键词（包括粗粒度和细粒度关键词）来评估检索到的上下文，而不是依赖于对黄金分块的标注。此外，我们发布了一个专为多样化数据场景量身定制的整体基准数据集，涵盖了广泛的文档格式和查询类型。我们通过实验评估了RAG系统的各个阶段，展示了CoFE-RAG框架的实用性。我们的评估方法为RAG系统在处理多样化数据场景时的有效性提供了独特的见解，使其对自身能力和局限性有了更细致的理解。
英文摘要：
Retrieval-Augmented Generation (RAG) aims to enhance large language models (LLMs) to generate more accurate and reliable answers with the help of the retrieved context from external knowledge sources, thereby reducing the incidence of hallucinations. Despite the advancements, evaluating these systems remains a crucial research area due to the following issues: (1) Limited data diversity: The insufficient diversity of knowledge sources and query types constrains the applicability of RAG systems; (2) Obscure problems location: Existing evaluation methods have difficulty in locating the stage of the RAG pipeline where problems occur; (3) Unstable retrieval evaluation: These methods often fail to effectively assess retrieval performance, particularly when the chunking strategy changes. To tackle these challenges, we propose a Comprehensive Full-chain Evaluation (CoFE-RAG) framework to facilitate thorough evaluation across the entire RAG pipeline, including chunking, retrieval, reranking, and generation. To effectively evaluate the first three phases, we introduce multi-granularity keywords, including coarse-grained and fine-grained keywords, to assess the retrieved context instead of relying on the annotation of golden chunks. Moreover, we release a holistic benchmark dataset tailored for diverse data scenarios covering a wide range of document formats and query types. We demonstrate the utility of the CoFE-RAG framework by conducting experiments to evaluate each stage of RAG systems. Our evaluation method provides unique insights into the effectiveness of RAG systems in handling diverse data scenarios, offering a more nuanced understanding of their capabilities and limitations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
