arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.12941cs.CL
收藏

事实、获取与推理:检索增强生成的一项统一评估

Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation

Harvard University(哈佛大学);Google DeepMind(谷歌DeepMind);Meta

Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui

AI总结:

提出FRAMES评估数据集,通过多跳问题统一评估LLMs在RAG场景下的事实性、检索与推理能力,并设计多步检索流水线将准确率从0.40提升至0.66。

摘要:

大型语言模型(LLMs)在各种认知任务中展现出了显著的性能提升。一个新兴的应用是利用LLMs来增强检索增强生成(RAG)能力。这些系统要求LLMs理解用户查询,检索相关信息,并合成连贯且准确的回答。鉴于此类系统在现实世界中的部署日益增加,全面的评估变得至关重要。为此,我们提出了FRAMES(Factuality, Retrieval, And reasoning MEasurement Set),这是一个高质量评估数据集,旨在测试LLMs提供事实性回答的能力,评估检索能力,并评估生成最终答案所需的推理能力。虽然先前的工作提供了孤立评估这些能力的数据集和基准,但FRAMES提供了一个统一框架,能够更清晰地展现LLMs在端到端RAG场景中的表现。我们的数据集包含具有挑战性的多跳问题,需要整合来自多个来源的信息。我们提供的基线结果表明,即使是当前最先进的LLMs在处理该任务时也面临困难,在无检索情况下仅达到0.40的准确率。通过我们提出的多步检索流水线,准确率得到了显著提升,达到了0.66的准确率(提升超过50%)。我们希望这项工作能帮助弥合评估差距,并协助开发更强大、更具能力的RAG系统。

英文摘要:

Large Language Models (LLMs) have demonstrated significant performance improvements across various cognitive tasks. An emerging application is using LLMs to enhance retrieval-augmented generation (RAG) capabilities. These systems require LLMs to understand user queries, retrieve relevant information, and synthesize coherent and accurate responses. Given the increasing real-world deployment of such systems, comprehensive evaluation becomes crucial. To this end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set), a high-quality evaluation dataset designed to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. While previous work has provided datasets and benchmarks to evaluate these abilities in isolation, FRAMES offers a unified framework that provides a clearer picture of LLM performance in end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions that require the integration of information from multiple sources. We present baseline results demonstrating that even state-of-the-art LLMs struggle with this task, achieving 0.40 accuracy with no retrieval. The accuracy is significantly improved with our proposed multi-step retrieval pipeline, achieving an accuracy of 0.66 (>50% improvement). We hope our work will help bridge evaluation gaps and assist in developing more robust and capable RAG systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑