arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.04744cs.CL
收藏

CRAG——综合性检索增强生成基准

CRAG -- Comprehensive RAG Benchmark

Meta;FAIR(Facebook人工智能研究院);HKUST(香港科技大学);HKUST (GZ)(香港科技大学(广州));Meta Reality Labs(元现实实验室)
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Mora… 展开作者

Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen-tau Yih, Xin Luna Dong

AI总结:

本研究推出包含4409个问答对的综合性RAG基准CRAG,覆盖多领域多类问题与动态性特征,评估发现现有LLM及RAG方案准确率仍有不足,该基准为相关研究及赛事提供支撑。

摘要:

检索增强生成(RAG)近期已成为缓解大语言模型(LLM)知识匮乏问题的一种颇具前景的解决方案。然而,现有的RAG数据集未能充分体现现实世界问答(QA)任务的多样性与动态性。为填补这一空白,我们推出了综合性RAG基准(Comprehensive RAG Benchmark,CRAG),这是一个包含4409个问答对以及用于模拟网页和知识图谱(KG)搜索的模拟应用程序接口(API)的事实性问答基准。CRAG的设计涵盖了五大领域、八大问题类别的各类问题,反映了从热门到长尾的不同实体流行度,以及从数年到数秒的时间动态性。我们对该基准的评估凸显了实现完全可信问答仍存在差距:大多数先进的大语言模型在CRAG上的准确率≤34%,直接添加RAG仅能将准确率提升至44%,而业界最先进的RAG解决方案也只能在无任何幻觉的情况下答对63%的问题。CRAG还揭示出,在回答动态性更强、流行度更低或复杂度更高的事实相关问题时,准确率要低得多,这为未来研究指明了方向。CRAG基准为KDD Cup 2024挑战赛奠定了基础,吸引了数千名参与者和参赛作品。我们承诺将持续维护CRAG,为研究界推进RAG解决方案及通用问答解决方案提供支持。CRAG可通过https://github.com/facebookresearch/CRAG/获取。

英文摘要:

Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets, however, do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a diverse array of questions across five domains and eight question categories, reflecting varied entity popularity from popular to long-tail, and temporal dynamisms ranging from years to seconds. Our evaluation of this benchmark highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63% of questions without any hallucination. CRAG also reveals much lower accuracy in answering questions regarding facts with higher dynamism, lower popularity, or higher complexity, suggesting future research directions. The CRAG benchmark laid the groundwork for a KDD Cup 2024 challenge and attracted thousands of participants and submissions. We commit to maintaining CRAG to serve research communities in advancing RAG solutions and general QA solutions. CRAG is available at https://github.com/facebookresearch/CRAG/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑