RAGalyst：面向特定领域RAG的自动化人本对齐智能体评估框架
RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAG
AI总结：
本文提出RAGalyst框架，通过智能体流水线生成高质量合成QA数据并优化LLM-as-a-Judge指标，实现与人类判断高度对齐的特定领域RAG自动化评估，揭示跨领域性能差异与设计权衡。
摘要：
检索增强生成（RAG）是将大语言模型（LLM）建立在事实证据之上的一项关键技术，然而在专业化、安全关键的领域中评估RAG系统仍然是一项重大挑战。现有评估框架通常依赖基于启发式的指标，无法捕捉特定领域的细微差别；其他工作则采用LLM-as-a-Judge方法，但缺乏经过验证的与人类判断的一致性。本文提出RAGalyst，一个自动化、人本对齐的智能体框架，旨在对特定领域RAG系统进行严格评估。RAGalyst具有一条智能体流水线，能够从源文档生成高质量、合成的问答（QA）数据集，并引入智能体过滤步骤以确保数据保真度。该框架通过提示优化对两个关键的LLM-as-a-Judge指标——答案正确性（Answer Correctness）和可回答性（Answerability）——进行精化，以实现与人类标注的高度相关性。将该框架应用于评估三个不同领域（军事行动、网络安全和桥梁工程）中的各种RAG组件，我们发现性能高度依赖于上下文。没有任何单一的嵌入模型、LLM或超参数配置被证明是普遍最优的。此外，我们还对RAG中最常见的低答案正确性原因进行了分析。这些发现凸显了像RAGalyst这样的系统性评估框架的必要性，它使从业者能够揭示特定领域的权衡，并为构建可靠且有效的RAG系统做出明智的设计选择。RAGalyst已在我们的Github上提供。
英文摘要：
Retrieval-Augmented Generation (RAG) is a critical technique for grounding Large Language Models (LLMs) in factual evidence, yet evaluating RAG systems in specialized, safety-critical domains remains a significant challenge. Existing evaluation frameworks often rely on heuristic-based metrics that fail to capture domain-specific nuances and other works utilize LLM-as-a-Judge approaches that lack validated alignment with human judgment. This paper introduces RAGalyst, an automated, human-aligned agentic framework designed for the rigorous evaluation of domain-specific RAG systems. RAGalyst features an agentic pipeline that generates high-quality, synthetic question-answering (QA) datasets from source documents, incorporating an agentic filtering step to ensure data fidelity. The framework refines two key LLM-as-a-Judge metrics-Answer Correctness and Answerability-using prompt optimization to achieve a strong correlation with human annotations. Applying this framework to evaluate various RAG components across three distinct domains (military operations, cybersecurity, and bridge engineering), we find that performance is highly context-dependent. No single embedding model, LLM, or hyperparameter configuration proves universally optimal. Additionally, we provide an analysis on the most common low Answer Correctness reasons in RAG. These findings highlight the necessity of a systematic evaluation framework like RAGalyst, which empowers practitioners to uncover domain-specific trade-offs and make informed design choices for building reliable and effective RAG systems. RAGalyst is available on our Github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
