CRMArena-Pro：跨多样业务场景与交互的LLM智能体整体评估
CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions
AI总结：
针对现有业务基准缺乏真实性和覆盖度的问题，提出CRMArena-Pro基准，含19个专家验证任务和多轮交互，发现领先LLM单轮成功率仅58%，多轮降至35%，保密意识近乎为零，凸显能力差距。
摘要：
尽管AI智能体在业务领域具有变革性潜力，但在广泛使用的平台上，缺乏公开、真实的业务数据阻碍了有效的性能基准测试。现有基准通常在环境、数据和智能体-用户交互方面缺乏保真度，且对多样化业务场景和行业的覆盖有限。为弥补这些不足，我们引入了CRMArena-Pro，这是一个新颖的基准，用于在多样化专业环境中对LLM智能体进行整体、真实的评估。CRMArena-Pro在CRMArena的基础上扩展了十九个经专家验证的任务，涵盖销售、服务以及“配置、定价和报价”流程，同时适用于企业对企业（B2B）和企业对客户（B2C）场景。它独特地融入了由多样化角色引导的多轮交互以及稳健的保密意识评估。实验表明，领先的LLM智能体在CRMArena-Pro上的单轮成功率仅为约58%，而在多轮设置中性能显著下降至约35%。虽然工作流执行对顶级智能体而言更为可行（单轮成功率超过83%），但其他被评估的业务技能则带来更大挑战。此外，智能体表现出近乎为零的内在保密意识；尽管有针对性的提示可以改善这一点，但往往会损害任务性能。这些发现凸显了当前LLM能力与企业需求之间的巨大差距，强调了在多轮推理、保密遵从和多技能获取方面取得进展的必要性。
英文摘要：
While AI agents hold transformative potential in business, effective performance benchmarking is hindered by the scarcity of public, realistic business data on widely used platforms. Existing benchmarks often lack fidelity in their environments, data, and agent-user interactions, with limited coverage of diverse business scenarios and industries. To address these gaps, we introduce CRMArena-Pro, a novel benchmark for holistic, realistic assessment of LLM agents in diverse professional settings. CRMArena-Pro expands on CRMArena with nineteen expert-validated tasks across sales, service, and 'configure, price, and quote' processes, for both Business-to-Business and Business-to-Customer scenarios. It distinctively incorporates multi-turn interactions guided by diverse personas and robust confidentiality awareness assessments. Experiments reveal leading LLM agents achieve only around 58% single-turn success on CRMArena-Pro, with performance dropping significantly to approximately 35% in multi-turn settings. While Workflow Execution proves more tractable for top agents (over 83% single-turn success), other evaluated business skills present greater challenges. Furthermore, agents exhibit near-zero inherent confidentiality awareness; though targeted prompting can improve this, it often compromises task performance. These findings highlight a substantial gap between current LLM capabilities and enterprise demands, underscoring the need for advancements in multi-turn reasoning, confidentiality adherence, and versatile skill acquisition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
