arXiv 2604.092852026-04-10cs.AI

SAGE：基于服务代理图的评估基准

SAGE: A Service Agent Graph-guided Evaluation Benchmark

Tianjin University（天津大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；The Chinese University of Hong Kong（香港中文大学）；Independent Researcher（独立研究员）

Ling Shi, Yuqin Dai, Ziyin Wang, Ning Gao, Wei Zhang, Chaozheng Wang, Yujie Wang, Wei He, Jinpeng Wang, Deiyi Xiong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SAGE基准，通过动态对话图验证逻辑合规性，解决现有基准无法评估多维度用户行为和SOP的问题，揭示模型在意图分类与后续动作推导间的执行差距。

摘要：

大型语言模型（LLM）的发展推动了客户服务自动化，但其性能评估仍具挑战。现有基准多依赖静态方法和单一指标，无法涵盖多样化的用户行为或实际部署中严格遵循的标准操作程序（SOP）。为此，我们提出SAGE（服务代理图引导评估），一种通用多代理基准，用于自动化双轴评估。SAGE将无结构SOP转化为动态对话图，以精确验证逻辑合规性和全面路径覆盖。我们引入对抗意图分类法和模块化扩展机制，实现跨领域的低成本部署，并促进自动对话数据生成。评估通过Judge Agents和规则引擎分析用户与服务代理的交互，生成确定性真实值。在27个LLM上进行的6种工业场景实验揭示了显著的“执行差距”，即模型能准确分类意图但无法推导正确后续动作。我们还观察到“共情韧性”现象，即在高对抗强度下，模型仍能维持礼貌的对话表象。代码和资源可在https://anonymous.4open.science/r/SAGE-Bench-4CD3/获取。

英文摘要：

The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominantly rely on static paradigms and single-dimensional metrics, failing to account for diverse user behaviors or the strict adherence to structured Standard Operating Procedures (SOPs) required in real-world deployments. To bridge this gap, we propose SAGE (Service Agent Graph-guided Evaluation), a universal multi-agent benchmark for automated, dual-axis assessment. SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance and comprehensive path coverage. We introduce an Adversarial Intent Taxonomy and a modular Extension Mechanism, enabling low-cost deployment across domains and facilitating automated dialogue data synthesis. Evaluation is conducted via a framework where Judge Agents and a Rule Engine analyze interactions between User and Service Agents to generate deterministic ground truth. Extensive experiments on 27 LLMs across 6 industrial scenarios reveal a significant ``Execution Gap'' where models accurately classify intents but fail to derive correct subsequent actions. We also observe ``Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures under high adversarial intensity. Code and resources are available at https://anonymous.4open.science/r/SAGE-Bench-4CD3/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。