超越IVR：评估符合业务规范的客户支持LLM代理的基准测试
Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence
AI总结：
本文提出JourneyBench基准测试，用于评估客户支持中遵循业务政策的LLM代理，展示动态提示代理在提升政策遵循度方面的有效性。
摘要：
传统客户支持系统，如交互式语音应答（IVR），依赖于僵化的脚本，缺乏处理复杂、政策驱动任务所需的灵活性。尽管大型语言模型（LLM）代理提供了有前景的替代方案，但评估其在遵循业务规则和现实世界支持工作流程方面的能力仍是一个开放性挑战。现有基准主要关注工具使用或任务完成，忽略了代理遵循多步骤政策、导航任务依赖性和在不可预测的用户或环境行为下保持稳健的能力。在本工作中，我们引入了JourneyBench，一个用于评估政策感知客户支持代理的基准。JourneyBench利用图表示来生成多样化、现实的客户支持场景，并提出用户旅程覆盖度分数，一种新的度量标准来衡量政策遵循程度。我们使用两种代理设计评估了多个最先进的LLM：静态提示代理（SPA）和动态提示代理（DPA），后者明确建模了政策控制。在三个领域中的703次对话中，我们显示DPA显著提高了政策遵循度，甚至允许较小的模型如GPT-4o-mini超越更强大的GPT-4o。我们的发现证明了结构化编排的重要性，并确立了JourneyBench作为推进AI驱动客户支持超越IVR时代限制的关键资源。
英文摘要：
Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent's capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
