RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework
Tsinghua University(清华大学);Beijing Normal University(北京师范大学);Northeastern University(东北大学);University of Chinese Academy of Sciences(中国科学院大学);ModelBest;University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
Retrieval-Augmented Generation (RAG) is a powerful approach that enables large language models (LLMs) to incorporate external knowledge. However, evaluating the effectiveness of RAG systems in specialized scenarios remains challenging due to the high costs of data construction and the lack of suitable evaluation metrics. This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios by generating high-quality documents, questions, answers, and references through a schema-based pipeline. With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance to evaluate LLM generated responses rigorously. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. Furthermore, the use of LLMs for scoring the proposed metrics demonstrates a high level of consistency with human evaluations. RAGEval establishes a new paradigm for evaluating RAG systems in real-world applications. The code and dataset are released at https://github.com/OpenBMB/RAGEval.
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
University of Science and Technology of China(中国科学技术大学);Institute for Advanced Algorithms Research (Shanghai)((上海)高级算法研究院);Renmin University of China(中国人民大学);AI Research Institute(360人工智能研究院)
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院);College of Design and Innovation, Tongji University(同济大学设计创意学院)
Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systems
School of Computer Science and Engineering, Southeast University, Nanjing 211189, China(计算机科学与工程学院,东南大学,南京211189,中国);Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用重点实验室(东南大学),教育部,中国);School of Law, Southeast University, Nanjing 211189, China(法学院,东南大学,南京211189,中国)
RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Xiaohongshu Inc.(小红书公司);Xi’an Jiaotong University(西安交通大学)
HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks
Peking University(北京大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);University of Science and Technology of China(中国科学技术大学);The Hong Kong University of Science and Technology(香港科技大学)
KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG
School of Biomedical Engineering, Tsinghua, China(清华大学生物医学工程学院);School of Informatics, Xiamen University, China(厦门大学信息学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学计算机科学与技术系);Dept. of Computer Science and Technology, Northeastern University, Shenyang, China(东北大学沈阳校区计算机科学与技术系);Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China(中国科学院信息工程研究所)
WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs
Ping An Bank Co., Ltd.(平安银行股份有限公司);Tsinghua University(清华大学);Central South University(中南大学);Shanghai Yiwo Information Technology Co., Ltd.(上海易沃信息技术有限公司);Kwangwoon University(光云大学)
Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation
National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);City University of Hong Kong(香港城市大学);Tencent Technology(腾讯科技);Peking University(北京大学)
Toward General Instruction-Following Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)
Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Computer Engineering Department, Sharif University of Technology(沙斐大学计算机工程系);College of Interdisciplinary Science and Technology, University of Tehran(德黑兰大学跨学科科学与技术学院);Computer Engineering Department, K.N. Toosi University of Technology(技术学院计算机工程系);Qatar Computing Research Institute(卡塔尔计算研究院)