arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.01511cs.AI
收藏

CRAB:面向多模态语言模型智能体的跨环境智能体基准

CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents

Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin, Jian… 展开作者

Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin, Jianbo Deng, Philip Torr, Bernard Ghanem, Guohao Li

AI总结:

本文提出首个跨环境智能体基准框架Crab,通过基于图的细粒度评估和高效任务构建机制,在桌面与手机共120项任务上评测多模态语言模型,其中GPT-4o单智能体完成率最高,达38.01%。

摘要:

自主智能体的发展日益依赖多模态语言模型(MLMs)在图形用户界面(GUI)环境中执行自然语言描述的任务,例如网站、台式计算机或移动电话。现有针对交互式环境中MLM智能体的基准受限于仅关注单一环境、缺乏详细且通用的评估方法,以及任务和评估器构建的复杂性。为克服这些局限,我们提出了Crab,这是首个支持跨环境任务的智能体基准框架,融合了基于图的细粒度评估方法以及高效的任务与评估器构建机制。该框架支持多种设备,并可借助Python接口轻松扩展到任何环境。利用Crab,我们构建了一个跨平台的Crab Benchmark-v0,包含计算机桌面和移动电话环境中的120项任务。我们在此基准上使用不同的单智能体和多智能体系统配置评估了四种先进的MLM。实验结果表明,采用GPT-4o的单智能体取得了最佳完成率,为38.01%。所有框架代码、智能体代码和任务数据集均已在 https://github.com/camel-ai/crab 公开提供。

英文摘要:

The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and the complexities of constructing tasks and evaluators. To overcome these limitations, we introduce Crab, the first agent benchmark framework designed to support cross-environment tasks, incorporating a graph-based fine-grained evaluation method and an efficient mechanism for task and evaluator construction. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. Leveraging Crab, we developed a cross-platform Crab Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments. We evaluated four advanced MLMs using different single and multi-agent system configurations on this benchmark. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%. All framework code, agent code, and task datasets are publicly available at https://github.com/camel-ai/crab.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑