FeB4RAG：在检索增强生成上下文中评估联邦搜索
FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation
AI总结：
本文提出了专为RAG框架内联邦搜索设计的新数据集FeB4RAG，包含790个查询和LLM相关性判断，并证明高质量联邦搜索系统优于朴素方法。
摘要：
联邦搜索系统聚合来自多个搜索引擎的结果，通过选择合适的来源来提升结果质量并契合用户意图。随着检索增强生成（RAG）流水线的日益普及，联邦搜索在从异构数据源中检索相关信息以生成有依据的回答方面发挥着关键作用。然而，现有的数据集（例如过去 TREC FedWeb 轨道中开发的数据集）早于 RAG 范式转变，缺乏对现代信息检索挑战的代表性。为弥补这一空白，我们提出了 FeB4RAG，这是一个专门为 RAG 框架内的联邦搜索设计的新颖数据集。该数据集源自广泛使用的 BEIR 基准测试集的 16 个子集合，包含 790 个专为聊天机器人应用定制的信息请求（类似于对话查询），以及每个资源返回的顶级结果和相关的 LLM 生成的相关性判断。此外，为支持该数据集的需求，我们展示了高质量的 RAG 联邦搜索系统相较于朴素联邦搜索方法在响应生成上的影响。我们通过对 RAG 流水线生成的答案进行并排定性比较来实现这一点。我们的数据集促进并支持了新联邦搜索方法的开发与评估，尤其是在 RAG 流水线的上下文中。
英文摘要：
Federated search systems aggregate results from multiple search engines, selecting appropriate sources to enhance result quality and align with user intent. With the increasing uptake of Retrieval-Augmented Generation (RAG) pipelines, federated search can play a pivotal role in sourcing relevant information across heterogeneous data sources to generate informed responses. However, existing datasets, such as those developed in the past TREC FedWeb tracks, predate the RAG paradigm shift and lack representation of modern information retrieval challenges. To bridge this gap, we present FeB4RAG, a novel dataset specifically designed for federated search within RAG frameworks. This dataset, derived from 16 sub-collections of the widely used \beir benchmarking collection, includes 790 information requests (akin to conversational queries) tailored for chatbot applications, along with top results returned by each resource and associated LLM-derived relevance judgements. Additionally, to support the need for this collection, we demonstrate the impact on response generation of a high quality federated search system for RAG compared to a naive approach to federated search. We do so by comparing answers generated through the RAG pipeline through a qualitative side-by-side comparison. Our collection fosters and supports the development and evaluation of new federated search methods, especially in the context of RAG pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
