RAGBench：面向检索增强生成系统的可解释基准
RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
AI总结：
针对RAG系统缺乏统一评估标准和标注数据集的问题，该研究推出含10万条样本、覆盖5个行业领域的RAGBench基准，提出TRACe可解释评估框架，发现微调RoBERTa评估效果优于LLM类方法，助力RAG评估技术发展。
摘要：
检索增强生成（Retrieval-Augmented Generation, RAG）已成为将领域特定知识整合到由大语言模型（Large Language Models, LLMs）驱动的面向用户的聊天应用中的标准架构模式。RAG系统的核心特征包括：（1）文档检索器，用于在领域特定语料库中查询与输入查询相关的上下文信息；（2）大语言模型，用于基于提供的查询和上下文生成响应。然而，由于缺乏统一的评估标准和标注数据集，RAG系统的综合评估仍是一项挑战。针对这一问题，我们推出了RAGBench：首个包含10万条样本的综合大规模RAG基准数据集，覆盖5个独特的行业特定领域和多种RAG任务类型。RAGBench的样本来源于用户手册等行业语料，因此与行业应用的相关性尤其突出。此外，我们形式化定义了TRACe评估框架：一套适用于所有RAG领域的、可解释且可落地的RAG评估指标。我们已在https://huggingface.co/datasets/rungalileo/ragbench 发布该标注数据集。RAGBench的可解释标签支持对RAG系统进行全面评估，能够为生产应用的持续迭代提供可落地的反馈。通过全面的基准测试，我们发现基于LLM的RAG评估方法在RAG评估任务上难以与经过微调的RoBERTa模型相匹敌。我们明确了现有方法的不足领域，并提出采用搭配TRACe框架的RAGBench，以推动RAG评估系统的技术发展。
英文摘要：
Retrieval-Augmented Generation (RAG) has become a standard architectural pattern for incorporating domain-specific knowledge into user-facing chat applications powered by Large Language Models (LLMs). RAG systems are characterized by (1) a document retriever that queries a domain-specific corpus for context information relevant to an input query, and (2) an LLM that generates a response based on the provided query and context. However, comprehensive evaluation of RAG systems remains a challenge due to the lack of unified evaluation criteria and annotated datasets. In response, we introduce RAGBench: the first comprehensive, large-scale RAG benchmark dataset of 100k examples. It covers five unique industry-specific domains and various RAG task types. RAGBench examples are sourced from industry corpora such as user manuals, making it particularly relevant for industry applications. Further, we formalize the TRACe evaluation framework: a set of explainable and actionable RAG evaluation metrics applicable across all RAG domains. We release the labeled dataset at https://huggingface.co/datasets/rungalileo/ragbench. RAGBench explainable labels facilitate holistic evaluation of RAG systems, enabling actionable feedback for continuous improvement of production applications. Thorough extensive benchmarking, we find that LLM-based RAG evaluation methods struggle to compete with a finetuned RoBERTa model on the RAG evaluation task. We identify areas where existing approaches fall short and propose the adoption of RAGBench with TRACe towards advancing the state of RAG evaluation systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
