AgencyBench：在100万token现实场景中自主代理的前沿评估
AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
AI总结：
本文提出AgencyBench，通过138个现实场景评估6种核心代理能力，揭示闭源模型在资源效率和反馈修正上的优势，为下一代自主代理的发展提供测试平台。
摘要：
大型语言模型（LLMs）基于的自主代理在经济生产中展现出多方面的能力。然而，现有基准测试仍局限于单一代理能力，未能捕捉长期现实场景。此外，依赖人工反馈进行真实任务的评估存在可扩展性瓶颈，阻碍了自动化评估的实施。为此，我们引入AgencyBench，一个基于日常AI使用场景的综合基准测试，评估6种核心代理能力，涵盖32个现实场景，包含138个任务，每个任务需要平均90次工具调用、100万token和数小时的执行时间。为了实现自动化评估，我们采用用户模拟代理提供迭代反馈，并使用Docker沙箱进行视觉和功能评估。实验表明，闭源模型显著优于开源模型（48.4% vs 32.1%）。进一步分析显示，不同模型在资源效率、反馈驱动的自我修正和特定工具使用偏好上存在显著差异。最后，我们研究了代理支架的影响，发现专有模型在其原生生态系统中表现更优（例如Claude-4.5-Opus通过Claude-Agent-SDK），而开源模型在特定执行框架中表现出不同的性能峰值。AgencyBench为下一代代理提供了关键测试平台，强调了模型架构与代理框架协同优化的必要性。我们相信这项工作为自主代理的未来方向提供了新的视角，并在https://github.com/GAIR-NLP/AgencyBench发布了完整的基准测试和评估工具包。
英文摘要：
Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. However, existing benchmarks remain focused on single agentic capability, failing to capture long-horizon real-world scenarios. Moreover, the reliance on human-in-the-loop feedback for realistic tasks creates a scalability bottleneck, hindering automated rollout collection and evaluation. To bridge this gap, we introduce AgencyBench, a comprehensive benchmark derived from daily AI usage, evaluating 6 core agentic capabilities across 32 real-world scenarios, comprising 138 tasks with specific queries, deliverables, and rubrics. These scenarios require an average of 90 tool calls, 1 million tokens, and hours of execution time to resolve. To enable automated evaluation, we employ a user simulation agent to provide iterative feedback, and a Docker sandbox to conduct visual and functional rubric-based assessment. Experiments reveal that closed-source models significantly outperform open-source models (48.4% vs 32.1%). Further analysis reveals significant disparities across models in resource efficiency, feedback-driven self-correction, and specific tool-use preferences. Finally, we investigate the impact of agentic scaffolds, observing that proprietary models demonstrate superior performance within their native ecosystems (e.g., Claude-4.5-Opus via Claude-Agent-SDK), while open-source models exhibit distinct performance peaks, suggesting potential optimization for specific execution frameworks. AgencyBench serves as a critical testbed for next-generation agents, highlighting the necessity of co-optimizing model architecture with agentic frameworks. We believe this work sheds light on the future direction of autonomous agents, and we release the full benchmark and evaluation toolkit at https://github.com/GAIR-NLP/AgencyBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
