生成式信息检索评估方法比较
A Comparison of Methods for Evaluating Generative IR
AI总结：
针对回答不再来自固定文档集合的 Gen-IR 系统，本文比较基于 LLM 标签、相关性、子主题、偏好和嵌入的离线评估方法，并通过 TREC 任务验证其与人工判断的一致性及可审计性。
摘要：
信息检索系统正越来越多地纳入生成式组件。例如，在检索增强生成（RAG）系统中，检索组件可能提供真值来源，而生成式组件对其回答进行总结和增强。在另一些系统中，大型语言模型（LLM）可能不咨询检索组件而直接生成回答。尽管生成式信息检索（Gen-IR）系统存在多种定义，但在本文中，我们关注那些系统回答并非取自固定文档或段落集合的系统。对查询的回答可能是全新文本。由于传统信息检索评估方法在这种模型下失效，我们探索了将传统离线评估方法扩展到 Gen-IR 场景的各种方法。离线信息检索评估传统上雇用付费人工评估员，但 LLM 正越来越多地取代人工评估，并表现出与众包标签相似或更优的能力。鉴于 Gen-IR 系统并非从固定集合生成回答，我们假设 Gen-IR 评估方法必须在很大程度上依赖 LLM 生成的标签。除基于二元相关性和分级相关性的方法外，我们还探索了基于显式子主题、成对偏好和嵌入的方法。我们首先在多项 TREC Deep Learning Track 任务上对照人工评估验证这些方法；随后将这些方法用于评估若干纯生成式系统的输出。对于每种方法，我们同时考察其在无需人工标签或其他输入情况下自主行动的能力，以及其支持人工审计的能力。为信任这些方法，我们必须确信其结果与人工评估一致。为此，评估标准必须透明，以便结果能够接受人工评估员审计。
英文摘要：
Information retrieval systems increasingly incorporate generative components. For example, in a retrieval augmented generation (RAG) system, a retrieval component might provide a source of ground truth, while a generative component summarizes and augments its responses. In other systems, a large language model (LLM) might directly generate responses without consulting a retrieval component. While there are multiple definitions of generative information retrieval (Gen-IR) systems, in this paper we focus on those systems where the system's response is not drawn from a fixed collection of documents or passages. The response to a query may be entirely new text. Since traditional IR evaluation methods break down under this model, we explore various methods that extend traditional offline evaluation approaches to the Gen-IR context. Offline IR evaluation traditionally employs paid human assessors, but increasingly LLMs are replacing human assessment, demonstrating capabilities similar or superior to crowdsourced labels. Given that Gen-IR systems do not generate responses from a fixed set, we assume that methods for Gen-IR evaluation must largely depend on LLM-generated labels. Along with methods based on binary and graded relevance, we explore methods based on explicit subtopics, pairwise preferences, and embeddings. We first validate these methods against human assessments on several TREC Deep Learning Track tasks; we then apply these methods to evaluate the output of several purely generative systems. For each method we consider both its ability to act autonomously, without the need for human labels or other input, and its ability to support human auditing. To trust these methods, we must be assured that their results align with human assessments. In order to do so, evaluation criteria must be transparent, so that outcomes can be audited by human assessors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
