检索增强生成(RAG)近期已成为缓解大语言模型(LLM)知识匮乏问题的一种颇具前景的解决方案。然而,现有的RAG数据集未能充分体现现实世界问答(QA)任务的多样性与动态性。为填补这一空白,我们推出了综合性RAG基准(Comprehensive RAG Benchmark,CRAG),这是一个包含4409个问答对以及用于模拟网页和知识图谱(KG)搜索的模拟应用程序接口(API)的事实性问答基准。CRAG的设计涵盖了五大领域、八大问题类别的各类问题,反映了从热门到长尾的不同实体流行度,以及从数年到数秒的时间动态性。我们对该基准的评估凸显了实现完全可信问答仍存在差距:大多数先进的大语言模型在CRAG上的准确率≤34%,直接添加RAG仅能将准确率提升至44%,而业界最先进的RAG解决方案也只能在无任何幻觉的情况下答对63%的问题。CRAG还揭示出,在回答动态性更强、流行度更低或复杂度更高的事实相关问题时,准确率要低得多,这为未来研究指明了方向。CRAG基准为KDD Cup 2024挑战赛奠定了基础,吸引了数千名参与者和参赛作品。我们承诺将持续维护CRAG,为研究界推进RAG解决方案及通用问答解决方案提供支持。CRAG可通过https://github.com/facebookresearch/CRAG/获取。
英文摘要:
Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets, however, do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a diverse array of questions across five domains and eight question categories, reflecting varied entity popularity from popular to long-tail, and temporal dynamisms ranging from years to seconds. Our evaluation of this benchmark highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63% of questions without any hallucination. CRAG also reveals much lower accuracy in answering questions regarding facts with higher dynamism, lower popularity, or higher complexity, suggesting future research directions. The CRAG benchmark laid the groundwork for a KDD Cup 2024 challenge and attracted thousands of participants and submissions. We commit to maintaining CRAG to serve research communities in advancing RAG solutions and general QA solutions. CRAG is available at https://github.com/facebookresearch/CRAG/.
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
University of Science and Technology of China(中国科学技术大学);Institute for Advanced Algorithms Research (Shanghai)((上海)高级算法研究院);Renmin University of China(中国人民大学);AI Research Institute(360人工智能研究院)
Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);McGill University(麦吉尔大学);Tencent Company(腾讯公司);Artificial Intelligence Research Institute, iFLYTEK Co., Ltd(人工智能研究院,iFLYTEK公司)
A Navigational Approach for Comprehensive RAG via Traversal over Proposition Graphs
Idiap Research Institute(日内瓦研究所);École Polytechnique Fédérale de Lausanne(联邦理工学院洛桑分校);University of Manchester(曼彻斯特大学);CRUK Manchester Institute(CRUK曼彻斯特研究所)
KnowShiftQA: How Robust are RAG Systems when Textbook Knowledge Shifts in K-12 Education?
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China(计算机科学与工程系,香港科技大学,香港特别行政区,中国);Department of Mechanical Engineering, The University of Tokyo, Tokyo, Japan(机械工程系,东京大学,东京,日本)
FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain
Key Laboratory of High Confidence Software Technologies, CS, Peking University, China(北京大学高可信软件技术重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);State Key Laboratory of Multimedia Information Processing, School of Computer Sciences, Peking University(北京大学多媒体信息处理国家重点实验室)
DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generation
School of Information Science and Technology, University of Science and Technology of China(信息科学与技术学院,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心)
FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院智能信息处理重点实验室);University of Chinese Academy of Sciences, China(中国科学院大学);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen), China(哈尔滨工业大学(深圳)计算机与智能信息研究院)
Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Computer Engineering Department, Sharif University of Technology(沙斐大学计算机工程系);College of Interdisciplinary Science and Technology, University of Tehran(德黑兰大学跨学科科学与技术学院);Computer Engineering Department, K.N. Toosi University of Technology(技术学院计算机工程系);Qatar Computing Research Institute(卡塔尔计算研究院)