arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.14531cs.CLcs.IR
收藏

LiveRAG:用于RAG评估的具有不同难度级别的多样化问答数据集

LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation

Technology Innovation Institute (TII)(技术创新研究所)

David Carmel, Simone Filice, Guy Horowitz, Yoelle Maarek, Alex Shtoff, Oren Somekh, Ran Tavory

AI总结:

针对RAG系统评估需求,提出包含895个合成问答的LiveRAG公开基准,源自SIGIR'2025挑战赛数据集并补充真实答案等信息,通过IRT模型得出难度等得分,可支持系统评估与RAG研究推进。

摘要:

随着检索增强生成(RAG)在生成式AI解决方案中日益突出,系统性评估其有效性的需求逐渐显现。我们引入LiveRAG基准,这是一个包含895个合成问题与答案的公开数据集,旨在支持对基于RAG的问答系统进行系统性评估。该合成基准源自SIGIR'2025 LiveRAG挑战赛期间使用的数据集,参赛者在严格时间限制下接受评估。它补充了挑战赛期间未向参赛者提供的信息,例如真实答案及其相关支持性主张(用于评估参赛者的答案)。此外,每个问题都关联了估计的难度和区分度得分,这些得分通过对参赛者响应应用项目反应理论(IRT)模型得出。我们的分析突显了该基准问题的多样性、广泛的难度级别及其在区分系统能力方面的实用性。LiveRAG基准有望帮助社区推进RAG研究、进行系统性评估并开发更稳健的问答系统。

英文摘要:

With Retrieval Augmented Generation (RAG) becoming more and more prominent in generative AI solutions, there is an emerging need for systematically evaluating their effectiveness. We introduce the LiveRAG benchmark, a publicly available dataset of 895 synthetic questions and answers designed to support systematic evaluation of RAG-based Q&A systems. This synthetic benchmark is derived from the one used during the SIGIR'2025 LiveRAG Challenge, where competitors were evaluated under strict time constraints. It is augmented with information that was not made available to competitors during the Challenge, such as the ground-truth answers, together with their associated supporting claims which were used for evaluating competitors' answers. In addition, each question is associated with estimated difficulty and discriminability scores, derived from applying an Item Response Theory model to competitors' responses. Our analysis highlights the benchmark's questions diversity, the wide range of their difficulty levels, and their usefulness in differentiating between system capabilities. The LiveRAG benchmark will hopefully help the community advance RAG research, conduct systematic evaluation, and develop more robust Q&A systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑