RAG 系统是否覆盖了关键内容？基于子问题覆盖度的评估与优化
Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage
AI总结：
本文提出基于子问题覆盖度的RAG评估框架，将子问题分为核心、背景和后续三类，发现商用答案引擎遗漏约50%核心子问题，并证明利用核心子问题可将RAG系统胜率提升至74%。
摘要：
评估检索增强生成（RAG）系统仍然具有挑战性，尤其是对于缺乏确定答案且需要覆盖多个子主题的开放式问题。在本文中，我们引入了一个基于子问题覆盖度的新型评估框架，用于衡量 RAG 系统在多大程度上解决了问题的不同方面。我们提出将问题分解为子问题，并将其分为三种类型——核心、背景和后续——以反映其作用和重要性。利用这一分类，我们引入了一种细粒度评估协议，能够深入揭示 RAG 系统的检索和生成特征，包括三个商用生成式答案引擎：You.com、Perplexity AI 和 Bing Chat。有趣的是，我们发现尽管所有答案引擎覆盖核心子问题的频率都高于背景或后续子问题，但它们仍然遗漏了约 50% 的核心子问题，揭示了明显的改进空间。此外，子问题覆盖度指标被证明对响应排序有效，与人类偏好标注相比达到了 82% 的准确率。最后，我们还证明利用核心子问题能够增强 RAG 系统中的检索和答案生成，相较于缺乏子问题的基线实现了 74% 的胜率。
英文摘要：
Evaluating retrieval-augmented generation (RAG) systems remains challenging, particularly for open-ended questions that lack definitive answers and require coverage of multiple sub-topics. In this paper, we introduce a novel evaluation framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We propose decomposing questions into sub-questions and classifying them into three types -- core, background, and follow-up -- to reflect their roles and importance. Using this categorization, we introduce a fine-grained evaluation protocol that provides insights into the retrieval and generation characteristics of RAG systems, including three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. Interestingly, we find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions, revealing clear opportunities for improvement. Further, sub-question coverage metrics prove effective for ranking responses, achieving 82% accuracy compared to human preference annotations. Lastly, we also demonstrate that leveraging core sub-questions enhances both retrieval and answer generation in a RAG system, resulting in a 74% win rate over the baseline that lacks sub-questions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
