arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.01262cs.CLcs.IR
收藏

RAGEval:场景特定的RAG评估数据集生成框架

RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework

Tsinghua University(清华大学);Beijing Normal University(北京师范大学);Northeastern University(东北大学);University of Chinese Academy of Sciences(中国科学院大学);ModelBest;University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, Maosong Sun

AI总结:

本文提出RAGEval框架,通过基于schema的流水线生成高质量数据,并引入完整性、幻觉和不相关性三个新指标,有效评估特定场景下的RAG系统,在清晰度等多方面优于零样本方法且与人类评估高度一致。

摘要:

检索增强生成(RAG)是一种使大语言模型(LLMs)能够结合外部知识的强大方法。然而,由于数据构建的高成本和合适评估指标的缺乏,评估RAG系统在特定场景中的有效性仍然具有挑战性。本文介绍了RAGEval,这是一个旨在通过基于schema的流水线生成高质量文档、问题、答案和参考,来评估跨多种场景的RAG系统的框架。以事实准确性为重点,我们提出了三个新颖的指标:完整性、幻觉和不相关性,以严格评估LLM生成的响应。实验结果表明,RAGEval在生成样本的清晰度、安全性、一致性和丰富性方面优于零样本和单样本方法。此外,使用LLMs对所提指标进行评分显示出与人类评估高度一致。RAGEval为评估实际应用中的RAG系统建立了新范式。代码和数据集发布在https://github.com/OpenBMB/RAGEval。

英文摘要:

Retrieval-Augmented Generation (RAG) is a powerful approach that enables large language models (LLMs) to incorporate external knowledge. However, evaluating the effectiveness of RAG systems in specialized scenarios remains challenging due to the high costs of data construction and the lack of suitable evaluation metrics. This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios by generating high-quality documents, questions, answers, and references through a schema-based pipeline. With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance to evaluate LLM generated responses rigorously. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. Furthermore, the use of LLMs for scoring the proposed metrics demonstrates a high level of consistency with human evaluations. RAGEval establishes a new paradigm for evaluating RAG systems in real-world applications. The code and dataset are released at https://github.com/OpenBMB/RAGEval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑