面向企业级RAG系统的案例感知LLM-as-a-Judge评估
Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems
AI总结：
本文提出面向企业级RAG系统的案例感知LLM-as-a-Judge评估框架，通过八个操作指标评估多轮工作流，揭示企业关键权衡，提升系统诊断清晰度和可操作性。
摘要：
企业检索增强生成（RAG）助手在多轮、基于案例的工作流中运行，如技术支持和IT运维，其中评估必须反映操作约束、结构化标识符（例如错误代码、版本）和解决流程。现有RAG评估框架主要针对基准式或单轮设置设计，往往无法捕捉企业特定的失败模式，如案例误识别、流程不匹配和跨轮次部分解决。我们提出了一种面向企业多轮RAG系统的案例感知LLM-as-a-Judge评估框架。该框架使用八个基于操作的指标评估每个轮次，这些指标区分检索质量、 grounding保真度、答案效用、精度完整性以及案例/流程匹配。一种严重性感知评分协议减少了评分膨胀并提高了异构企业案例的诊断清晰度。系统使用确定性提示和严格JSON输出，实现了可扩展的批量评估、回归测试和生产监控。通过对比研究两个指令微调模型在短流程和长流程中的表现，我们表明通用代理指标提供模糊信号，而所提框架揭示了企业关键的权衡，这对系统改进具有可操作性。
英文摘要：
Enterprise Retrieval-Augmented Generation (RAG) assistants operate in multi-turn, case-based workflows such as technical support and IT operations, where evaluation must reflect operational constraints, structured identifiers (e.g., error codes, versions), and resolution workflows. Existing RAG evaluation frameworks are primarily designed for benchmark-style or single-turn settings and often fail to capture enterprise-specific failure modes such as case misidentification, workflow misalignment, and partial resolution across turns. We present a case-aware LLM-as-a-Judge evaluation framework for enterprise multi-turn RAG systems. The framework evaluates each turn using eight operationally grounded metrics that separate retrieval quality, grounding fidelity, answer utility, precision integrity, and case/workflow alignment. A severity-aware scoring protocol reduces score inflation and improves diagnostic clarity across heterogeneous enterprise cases. The system uses deterministic prompting with strict JSON outputs, enabling scalable batch evaluation, regression testing, and production monitoring. Through a comparative study of two instruction-tuned models across short and long workflows, we show that generic proxy metrics provide ambiguous signals, while the proposed framework exposes enterprise-critical tradeoffs that are actionable for system improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
