arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.11201cs.CLcs.AI
收藏

CocoaBench:评估真实世界的统一数字代理

CocoaBench: Evaluating Unified Digital Agents in the Wild

CocoaBench Team(CocoaBench 团队)
CocoaBench Team, Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, Haoxiang Zhang, Junli Wang… 展开作者

CocoaBench Team, Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, Haoxiang Zhang, Junli Wang, Hexi Jin, Boyuan Zheng, Kun Zhou, Yu Wang, Feng Yao, Licheng Liu, Yijiang Li, Zhifei Li, Zhengtao Han, Pracha Promthaw, Tommaso Cerruti, Xiaohan Fu, Ziqiao Ma, Jingbo Shang, Lianhui Qin, Julian McAuley, Eric P. Xing, Zhengzhong Liu, Rupesh Kumar Srivastava, Zhiting Hu

AI总结:

CocoaBench评估统一数字代理在多样化场景中的表现,通过人类设计的长周期任务测试其视觉、搜索和编码能力的灵活组合,发现当前代理在推理、规划和视觉理解方面仍有较大提升空间。

摘要:

如今,大型语言模型代理在软件工程、深度研究、GUI自动化等应用中表现强劲,但现有代理框架和模型日益将这些能力整合到统一系统中。然而,大多数评估仍单独测试这些能力,无法覆盖需要代理整合不同能力的多样化用例。我们引入CocoaBench,一个基于人类设计的长周期任务的基准,这些任务要求灵活组合视觉、搜索和编码能力。任务仅通过指令和自动评估函数定义,使评估在不同代理架构中可靠且可扩展。我们还提出了CocoaAgent,一个轻量级的共享框架,用于在不同模型基础上进行受控比较。实验表明,当前代理在CocoaBench上仍远未可靠,最佳评估系统仅达到45.1%的成功率。我们的分析进一步指出,在推理、规划、工具使用和视觉理解方面仍有显著提升空间。

英文摘要:

LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑