Long$^2$RAG：使用 Key Point Recall 评估长上下文与长格式的检索增强生成
Long$^2$RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall
AI总结：
针对现有RAG评估基准在长上下文检索与长格式生成评估上的不足，提出Long$^2$RAG基准及Key Point Recall指标，通过280个跨域问题与细致的关键点召回评估，更全面地衡量LLMs利用检索信息的能力。
摘要：
检索增强生成（RAG）是解决大语言模型（LLMs）中固定知识局限性的一种有前景的方法。然而，当前用于评估 RAG 系统的基准存在两个关键缺陷：（1）由于缺乏反映检索文档特征的数据集，它们未能充分衡量 LLMs 处理长上下文检索的能力；（2）它们缺乏一种综合评估方法，以评估 LLMs 生成有效利用检索信息的长格式回复的能力。为了解决这些不足，我们引入了 Long$^2$RAG 基准和 Key Point Recall (KPR) 指标。Long$^2$RAG 包含 280 个问题，跨越 10 个领域和 8 个问题类别，每个问题关联 5 篇检索文档，平均长度为 2,444 词。KPR 评估 LLMs 将从检索文档中提取的关键点纳入其生成回复的程度，为其利用检索信息的能力提供了更细致的评估。
英文摘要：
Retrieval-augmented generation (RAG) is a promising approach to address the limitations of fixed knowledge in large language models (LLMs). However, current benchmarks for evaluating RAG systems suffer from two key deficiencies: (1) they fail to adequately measure LLMs' capability in handling long-context retrieval due to a lack of datasets that reflect the characteristics of retrieved documents, and (2) they lack a comprehensive evaluation method for assessing LLMs' ability to generate long-form responses that effectively exploits retrieved information. To address these shortcomings, we introduce the Long$^2$RAG benchmark and the Key Point Recall (KPR) metric. Long$^2$RAG comprises 280 questions spanning 10 domains and across 8 question categories, each associated with 5 retrieved documents with an average length of 2,444 words. KPR evaluates the extent to which LLMs incorporate key points extracted from the retrieved documents into their generated responses, providing a more nuanced assessment of their ability to exploit retrieved information.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
