With Retrieval Augmented Generation (RAG) becoming more and more prominent in generative AI solutions, there is an emerging need for systematically evaluating their effectiveness. We introduce the LiveRAG benchmark, a publicly available dataset of 895 synthetic questions and answers designed to support systematic evaluation of RAG-based Q&A systems. This synthetic benchmark is derived from the one used during the SIGIR'2025 LiveRAG Challenge, where competitors were evaluated under strict time constraints. It is augmented with information that was not made available to competitors during the Challenge, such as the ground-truth answers, together with their associated supporting claims which were used for evaluating competitors' answers. In addition, each question is associated with estimated difficulty and discriminability scores, derived from applying an Item Response Theory model to competitors' responses. Our analysis highlights the benchmark's questions diversity, the wide range of their difficulty levels, and their usefulness in differentiating between system capabilities. The LiveRAG benchmark will hopefully help the community advance RAG research, conduct systematic evaluation, and develop more robust Q&A systems.
Athens University of Economics and Business(雅典经济与商业大学);Archimedes, Athena Research Center(阿基米德、雅典研究中心);Computer Science and Engineering, University of Ioannina(计算机科学与工程,伊奥安纳大学)
Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries
Yale University(耶鲁大学);University of Pennsylvania(宾夕法尼亚大学);Human Language Technology Center of Excellence(语言技术中心卓越中心);Johns Hopkins University(约翰霍普金斯大学)
Tackling the Inherent Difficulty of Noise Filtering in RAG
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室);School of Software Tsinghua University(清华大学软件学院)
LabourCrew: A Multi-Agent RAG Framework for Trustworthy Adversarial Deliberation and Statutory Reasoning over Labour Law
Ahsanullah University of Science and Technology(阿赫萨努拉科技大学);Jashore University of Science and Technology(杰肖尔科技大学);American International University - Bangladesh(美国国际大学-孟加拉)
SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context
School of Computer Science, University of Science and Technology of China(中国科学技术大学计算机科学学院);School of Biomedical Engineering, USTC(USTC生物医学工程学院);Suzhou Institute for Advanced Reasearch, USTC(USTC苏州市先进研究所)
DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generation
School of Information Science and Technology, University of Science and Technology of China(信息科学与技术学院,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心)
Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learning
McGill University \& Mila Montréal QC Canada;The Chinese University of Hong Kong Hong Kong China;Huawei Noah’s Ark Lab Montréal QC Canada;University of Manitoba Winnipeg MB Canada;Tianjin University Tianjin China;Independent Researcher Hong Kong China;McGill University \& Mila;The Chinese University of Hong Kong;Huawei Noah’s Ark Lab;University of Manitoba;Tianjin University;Independent Researcher
Generating Leakage-Free Benchmarks for Robust RAG Evaluation
Department of Computer Science, Purdue University(普渡大学计算机科学系);New Jersey Institute of Technology(新泽西理工学院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Microsoft Research(微软研究院)
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)
Rethinking Soft Compression in Retrieval-Augmented Generation: A Query-Conditioned Selector Perspective
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence(上海数据科学 key 实验室,计算机科学与人工智能学院);Fudan University(复旦大学);Shanghai(上海);China(中国)
CRISS: A Retrieval-Augmented AI Chatbot for Assisting Cancer Registrars
University of Missouri(密苏里大学);Missouri Cancer Registry and Research Center(密苏里癌症登记与研究中心);MU Institute for Data Science and Informatics(密苏里大学数据科学与信息学研究所)
UR$^2$: Unify RAG and Reasoning through Reinforcement Learning
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究机构(AIR),清华大学,北京,中国);School of Management Science & Information Engineering, Hebei University of Economics and Business, Hebei, China(管理科学与信息工程学院,河北经贸大学,河北,中国)
DreamerNLplus: Interpretable Modeling of Mental Health Dynamics from Social Media Timelines using Hybrid Rule-Based and RAG Methods
Leiden Institute of Advanced Computer Science (LIACS), Leiden University(莱顿高级计算机科学研究所(LIACS),莱顿大学);School of Computing and Communications (SCC), Lancaster University(计算与通信学院(SCC),兰卡斯特大学);Department of Medical Informatics, Erasmus University Medical Center Rotterdam(医学信息学系,埃因霍温医学中心鲁特万分校);Biomedical Data Sciences, Leiden University Medical Center(生物医学数据科学,莱顿大学医学中心)
GOSU: Retrieval-Augmented Generation with Global-Level Optimized Semantic Unit-Centric Framework
School of Future Science and Engineering, Soochow University(未来科学与工程学院,苏州大学);School of Mathematical Sciences, Soochow University(数学学院,苏州大学);School of Information Technology (Smart Campus Education Center), Suzhou Institute of Trade & Commerce(信息技术学院(智能校园教育中心),苏州贸易与商业学院)
Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation
MoE Key Lab of Artificial Intelligence(人工智能增强实验室);Jiangsu Key Lab of Language Computing(江苏语言计算重点实验室);X-LANCE Lab(X-LANCE实验室);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Shanghai Innovation Institute(上海创新研究院);Shanghai Aviation Electric Co., Ltd(上海航空电器有限公司)
Document-Level Numerical Reasoning across Single and Multiple Tables in Financial Reports
Department of Computer Science and Information Engineering, National Taiwan University(国立台湾大学资讯工程学系);FinTech Center, National Taiwan University(国立台湾大学金融科技中心)
Think Parallax: Solving Multi-Hop Problems via Multi-View Knowledge-Graph-Based Retrieval-Augmented Generation
Yangtze Delta Region Institute (Huzhou), University of Electronic Science and Technology of China(长江三角洲地区研究院(湖州),电子科学与技术大学);School of Information and Software Engineering, University of Electronic Science and Technology of China(信息与软件工程学院,电子科学与技术大学)
Exploration of Augmentation Strategies in Multi-modal Retrieval-Augmented Generation for the Biomedical Domain: A Case Study Evaluating Question Answering in Glycobiology
University of Maribor, Faculty of Health Sciences(莫拉维亚大学健康科学学院);University of Ljubljana, Medical Factory(卢布尔雅那大学医疗工厂);Genos Ltd(基因公司);Center for Smart Health, School of Nursing The Hong Kong Polytechnic University(智能健康中心护理学院香港理工大学);University of Zagreb, Faculty of Pharmacy(扎格雷布大学药学院);Usher Institute University of Edinburgh(埃德蒙顿大学usher研究所)
CLAIR-Fin: An Adversarial Multi-Agent Framework for Claim-Level Verification and Adaptive Debate in Cross-Modal Financial QA
Ahsanullah University of Science and Technology(阿萨努拉科技大学);Jashore University of Science and Technology(杰索尔科技大学);American International University - Bangladesh(孟加拉国美国国际大学)
Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室)