RAGChecker：一种用于诊断检索增强生成的细粒度框架
RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation
AI总结：
针对RAG系统因模块化、长文本评估等导致全面评估困难的问题，提出细粒度诊断框架RAGChecker，配备检索与生成模块诊断指标，相关性优于现有指标，可指导RAG系统开发，已开源。
摘要：
尽管检索增强生成（Retrieval-Augmented Generation，RAG）在利用外部知识方面展现出可观的能力，但由于RAG的模块化特性、长文本回复的评估难度以及测量结果的可靠性问题，对RAG系统开展全面评估仍颇具挑战。本文提出了细粒度评估框架RAGChecker，该框架为检索模块和生成模块配备了一套诊断指标。元评估验证表明，RAGChecker与人类判断的相关性显著优于其他评估指标。我们使用RAGChecker评估了8个RAG系统，并对其性能开展了深入分析，揭示了RAG架构设计选择中值得关注的模式与权衡关系。RAGChecker的指标能够指导研究人员与从业者开发更高效的RAG系统。该工作已在https://github.com/amazon-science/RAGChecker开源。
英文摘要：
Despite Retrieval-Augmented Generation (RAG) showing promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems. This work has been open sourced at https://github.com/amazon-science/RAGChecker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
