AgentCE-Bench: 一种可配置的智能体评估基准,具备可扩展的视野和可控难度,在轻量环境中
AgentCE-Bench: Agent Configurable Evaluation with Scalable Horizons and Controllable Difficulty under Lightweight Environments
AI总结:
AgentCE-Bench通过可扩展视野和可控难度,解决现有基准的高交互开销和任务分布不平衡问题,提供可靠、可解释的智能体评估。
摘要:
现有智能体基准存在两个关键限制:高环境交互开销(高达41%的总评估时间)和任务视野和难度分布不平衡,导致聚合分数不可靠。为此,我们提出AgentCE-Bench,围绕统一的网格规划任务,智能体需在部分完成的调度中填充隐藏槽位,受局部和全局约束。该基准通过两个正交轴提供精细控制:可扩展视野(通过隐藏槽位数H控制)和可控难度(通过解谜预算B决定全局误导候选数)。关键在于所有工具调用均通过静态JSON文件在轻量环境中解决,消除设置开销,实现快速、可重复的评估,适合训练时验证。我们首先验证H和B对任务视野和难度的可靠控制,以及AgentCE-Bench在领域一致性和模型区分性上的表现。然后在13种不同大小和家族的模型上进行跨6个领域的综合实验,揭示显著的跨模型性能差异,并确认AgentCE-Bench提供可解释和可控的智能体推理评估。
英文摘要:
Existing Agent benchmarks suffer from two critical limitations: high environment interaction overhead (up to 41\% of total evaluation time) and imbalanced task horizon and difficulty distributions that make aggregate scores unreliable. To address these issues, we propose AgentCE-Bench built around a unified grid-based planning task, where agents must fill hidden slots in a partially completed schedule subject to both local slot constraints and global constraints. Our benchmark offers fine-grained control through two orthogonal axes: \textbf{Scalable Horizons}, controlled by the number of hidden slots $H$, and \textbf{Controllable Difficulty}, governed by a decoy budget $B$ that determines the number of globally misleading decoy candidates. Crucially, all tool calls are resolved via static JSON files under a \textbf{Lightweight Environment} design, eliminating setup overhead and enabling fast, reproducible evaluation suitable for training-time validation. We first validate that $H$ and $B$ provide reliable control over task horizon and difficulty, and that AgentCE-Bench exhibits strong domain consistency and model discriminability. We then conduct comprehensive experiments across 13 models of diverse sizes and families over 6 domains, revealing significant cross-model performance variation and confirming that AgentCE-Bench provides interpretable and controllable evaluation of agent reasoning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。