生成用于RAG评估的多样化问答基准：DataMorgana
Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana
AI总结：
DataMorgana是一种生成高度可定制且多样化合成问答基准的工具，通过轻量级两阶段流程控制用户和问题类别分布，在词汇、句法和语义多样性上超越现有方法，用于RAG系统评估。
摘要：
评估检索增强生成（RAG）系统，尤其是在特定领域背景下，需要能够满足应用场景独特要求的基准。由于真实数据难以获取，一种常见策略是使用基于LLM的方法生成合成数据。现有解决方案是通用型的：给定一篇文档，它们生成一个问题以构建问答对。然而，尽管生成的问题个体质量可能不错，但它们通常不够多样化，无法合理覆盖真实最终用户与RAG系统交互的不同方式。我们在此介绍DataMorgana，一个用于生成高度可定制且多样化的合成问答基准的工具，专为RAG应用量身定制。DataMorgana支持对用户和问题类别进行详细配置，并控制其在基准中的分布。它采用轻量级两阶段流程，确保高效和快速迭代，同时生成反映预期流量分布的基准。我们进行了一系列全面的实验，定量和定性结果表明，DataMorgana在生成跨领域特定和通用知识语料库的词汇、句法和语义多样化问题集方面超越了现有工具和方法。DataMorgana将提供给研究社区中的选定团队作为首批测试者，用于即将在2025年2月初公布的SIGIR'2025 LiveRAG挑战赛。
英文摘要：
Evaluating Retrieval-Augmented Generation (RAG) systems, especially in domain-specific contexts, requires benchmarks that address the distinctive requirements of the applicative scenario. Since real data can be hard to obtain, a common strategy is to use LLM-based methods to generate synthetic data. Existing solutions are general purpose: given a document, they generate a question to build a Q&A pair. However, although the generated questions can be individually good, they are typically not diverse enough to reasonably cover the different ways real end-users can interact with the RAG system. We introduce here DataMorgana, a tool for generating highly customizable and diverse synthetic Q&A benchmarks tailored to RAG applications. DataMorgana enables detailed configurations of user and question categories and provides control over their distribution within the benchmark. It uses a lightweight two-stage process, ensuring efficiency and fast iterations, while generating benchmarks that reflect the expected traffic. We conduct a thorough line of experiments, showing quantitatively and qualitatively that DataMorgana surpasses existing tools and approaches in producing lexically, syntactically, and semantically diverse question sets across domain-specific and general-knowledge corpora. DataMorgana will be made available to selected teams in the research community, as first beta testers, in the context of the upcoming SIGIR'2025 LiveRAG challenge to be announced in early February 2025.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
