大型语言模型(LLMs)在各种认知任务中展现出了显著的性能提升。一个新兴的应用是利用LLMs来增强检索增强生成(RAG)能力。这些系统要求LLMs理解用户查询,检索相关信息,并合成连贯且准确的回答。鉴于此类系统在现实世界中的部署日益增加,全面的评估变得至关重要。为此,我们提出了FRAMES(Factuality, Retrieval, And reasoning MEasurement Set),这是一个高质量评估数据集,旨在测试LLMs提供事实性回答的能力,评估检索能力,并评估生成最终答案所需的推理能力。虽然先前的工作提供了孤立评估这些能力的数据集和基准,但FRAMES提供了一个统一框架,能够更清晰地展现LLMs在端到端RAG场景中的表现。我们的数据集包含具有挑战性的多跳问题,需要整合来自多个来源的信息。我们提供的基线结果表明,即使是当前最先进的LLMs在处理该任务时也面临困难,在无检索情况下仅达到0.40的准确率。通过我们提出的多步检索流水线,准确率得到了显著提升,达到了0.66的准确率(提升超过50%)。我们希望这项工作能帮助弥合评估差距,并协助开发更强大、更具能力的RAG系统。
英文摘要:
Large Language Models (LLMs) have demonstrated significant performance improvements across various cognitive tasks. An emerging application is using LLMs to enhance retrieval-augmented generation (RAG) capabilities. These systems require LLMs to understand user queries, retrieve relevant information, and synthesize coherent and accurate responses. Given the increasing real-world deployment of such systems, comprehensive evaluation becomes crucial. To this end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set), a high-quality evaluation dataset designed to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. While previous work has provided datasets and benchmarks to evaluate these abilities in isolation, FRAMES offers a unified framework that provides a clearer picture of LLM performance in end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions that require the integration of information from multiple sources. We present baseline results demonstrating that even state-of-the-art LLMs struggle with this task, achieving 0.40 accuracy with no retrieval. The accuracy is significantly improved with our proposed multi-step retrieval pipeline, achieving an accuracy of 0.66 (>50% improvement). We hope our work will help bridge evaluation gaps and assist in developing more robust and capable RAG systems.
RAR$^2$: Retrieval-Augmented Medical Reasoning via Thought-Driven Retrieval
Department of Computing, The Hong Kong Polytechnic University(计算机系,香港理工大学);Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology(可信自主系统研究院和计算机科学与工程系,南方科技大学)
CheckRLM: Effective Knowledge-Thought Coherence Checking in Retrieval-Augmented Reasoning
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);Beijing Key Laboratory of Artificial Intelligence for Education(北京人工智能教育重点实验室);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University(清华大学人工智能研究院计算机科学与技术系);Northeastern University(东北大学)
Detecting Hallucinations in Retrieval-Augmented Generation via Semantic-level Internal Reasoning Graph
Wangxuan Institute of Computer Technology, Peking University(王轩计算机技术研究所,北京大学);Institute of Artificial Intelligence and Future Networks, Beijing Normal University(人工智能与未来网络研究院,北京师范大学)
Data Augmented Pipeline for Legal Information Extraction and Reasoning
Center for Juris-Informatics, ROIS-DS(司法信息中心,ROIS-DS);Japan Advanced Institute of Science and Technology(日本先进科学研究所);Research and Development Center for LLMs, NII(大语言模型研发中心,NII)
KnowRA: Knowledge Retrieval Augmented Method for Document-level Relation Extraction with Comprehensive Reasoning Abilities
Nanjing Normal University, School of Computer Science and Electronic Informatics(南京师范大学计算机科学与电子信息学院);State Key Laboratory for Novel Software Technology at Nanjing University(南京大学新型软件技术国家重点实验室);Nanjing University, School of Computer Science(南京大学计算机科学学院)
HopRAG: Multi-Hop Reasoning for Logic-Aware Retrieval-Augmented Generation
Peking University(北京大学);Center for LLM, Institute for Advanced Algorithms Research(大模型中心,高级算法研究所);Huazhong University of Science and Technology(华中科技大学)
VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation
School of Software and Microelectronics, Peking University, China(北京大学软件与微电子学院);School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学人工智能研究院计算机科学与技术系)
Think Parallax: Solving Multi-Hop Problems via Multi-View Knowledge-Graph-Based Retrieval-Augmented Generation
Yangtze Delta Region Institute (Huzhou), University of Electronic Science and Technology of China(长江三角洲地区研究院(湖州),电子科学与技术大学);School of Information and Software Engineering, University of Electronic Science and Technology of China(信息与软件工程学院,电子科学与技术大学)
Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation
School of Computer Science & Tech, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of AI and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院)
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)
Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models
Institute for Artificial Intelligence, Tsinghua University (THUAI)(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology (BNRist)(北京信息科学与技术国家研究中心);Department of Automation, Tsinghua University(清华大学自动化系);TMLR Group, Hong Kong Baptist University(香港浸会大学TMLR组)
Knowledge Integration Decay in Search-Augmented Reasoning of Large Language Models
Department of Electrical and Computer Engineering, Seoul National University, Seoul, Korea(电气与计算机工程系,首尔国立大学,韩国首尔);Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Korea(人工智能交叉学科项目,首尔国立大学,韩国首尔)
Fast, Slow, and Tool-augmented Thinking for LLMs: A Review
School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China(上海交通大学机械工程学院);School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China(上海交通大学计算机科学学院);Huawei Technologies Co., Ltd., Beijing 100084, China(华为技术有限公司);Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong 999077, China(香港中文大学系统工程与工程管理系);Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong 999077, China(香港中文大学计算机科学与工程系);Huawei Hong Kong Research Center, Hong Kong 999077, China(华为香港研究中心)
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
Kuaishou Technology Co., Ltd.(快手科技有限公司);Institute for Clarity in Documentation(文献清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康分所);Rajiv Gandhi University(拉吉夫甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)