arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.32004cs.HCcs.CL
收藏

CRM-Agent-Bench:面向有状态企业环境的工具使用智能体基准测试

CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments

Salesforce, Inc.(赛富时有限公司)

Monojit Banerjee, Zhujun Lan, Defu Cao

AI总结:

本文提出CRM-Agent-Bench基准测试,评估16个智能体系统在555个CRM相关有状态多轮任务上的表现,发现领先系统严格成功率达64.32%-83.60%,但成本和各领域可靠性差异显著。

摘要:

将语言智能体部署到实际应用中,不仅需要给出正确答案,还要求智能体正确执行多步骤业务流程、遵循领域特定规则,并在对话中与用户交互,“说与做之间的差距”决定了客户满意还是造成实际损失。本文认为,可部署智能体的基准测试应联合评估四个方面:智能体达成的结果、执行轨迹、操作可行性以及策略合规性。客户关系管理(CRM)领域涉及可变记录、操作规则和相应影响,可全面体现这四个方面。本文提出CRM-Agent-Bench,这是一个基于执行的基准测试,包含9个业务领域、36个能力标签的555个有状态多轮任务。每个任务提供特定于任务的可执行契约,涵盖对操作、参数、顺序、响应、目标状态和受保护状态的检查;成功是该任务声明的检查的严格合取。本文通过这些契约评估完整智能体系统,报告得分覆盖率、延迟和成本。在对16个系统的审计最终版本评估中,严格成功率为64.32%至83.60%;尽管领先系统的表现接近,但它们在成本和各领域可靠性方面存在显著差异。CRM-Agent-Bench提供了一个可审计框架,用于测量基于契约的业务交易完整性和对任务声明的基准策略的遵守情况,其可移植结构设计为随着模型改进而扩展。Github仓库:this https URL

英文摘要:

Deploying language agents in real applications demands more than the right answer: an agent must execute a multi-step business process correctly, follow domain-specific rules, and interact with a user across a dialogue, where the gap between saying and doing is the gap between a satisfied customer and a real loss. We argue that a deployable-agent benchmark should jointly assess four facets: the outcome an agent reaches, the trajectory it takes, its operational viability, and its policy compliance. Customer relationship management (CRM), where mutable records meet operational rules and consequential effects, exercises all four. We introduce CRM-Agent-Bench, an execution-grounded benchmark of 555 stateful, multi-turn tasks across nine business domains and 36 capability tags. Each task supplies a task-specific executable contract drawn from checks over actions, arguments, order, response, target state, and protected state; success is the strict conjunction of the checks declared for that task. We evaluate the complete agent system through these contracts and report scored coverage, latency, and cost. Across sixteen systems evaluated on the audited final release, strict success spans 64.32% to 83.60%; the closely grouped leading systems nonetheless diverge sharply in cost and per-domain reliability. CRM-Agent-Bench offers an auditable framework for measuring contract-grounded business-transaction integrity and adherence to task-declared benchmark policies, with a portable structure designed for extension as models improve. Github repository: https://github.com/salesforce-misc/crm-agent-bench

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑