arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.07972cs.AIcs.CL
收藏

OSWorld:在真实计算机环境中针对开放式任务对多模态智能体进行基准测试

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

The University of Hong Kong(香港大学);CMU(卡内基梅隆大学);Salesforce Research(Salesforce研究院);University of Waterloo(滑铁卢大学)
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, … 展开作者

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu

AI总结:

针对现有基准无法反映真实计算机使用复杂性的问题,提出首个跨多操作系统的真实计算机环境OSWorld,并构建了369项开放式任务基准,评估发现最先进模型成功率远低于人类,为多模态通用智能体开发提供了重要见解。

摘要:

以最少人工干预完成复杂计算机任务的自主智能体,有望变革人机交互,显著提升可访问性与生产力。然而,现有基准测试要么缺乏交互式环境,要么局限于特定应用或领域的环境,无法反映真实世界计算机使用的多样性与复杂性,从而限制了任务范围与智能体可扩展性。为解决此问题,我们引入了OSWorld,这是首个同类可扩展的真实计算机环境,面向多模态智能体,支持跨Ubuntu、Windows和macOS等多种操作系统的任务设置、基于执行的评估和交互式学习。OSWorld可作为统一、集成的计算机环境,用于评估涉及任意应用程序的开放式计算机任务。基于OSWorld,我们创建了一个包含369项计算机任务的基准测试,涵盖开放领域中的真实网页与桌面应用、操作系统文件I/O,以及跨多个应用程序的工作流。每个任务示例均源自真实世界的计算机使用案例,并包含详细的初始状态设置配置和自定义的基于执行的评估脚本,以实现可靠、可复现的评估。在OSWorld上对最先进的基于LLM/VLM的智能体进行的广泛评估揭示了它们作为计算机助手方面的显著缺陷。人类能够完成超过72.36%的任务,而最佳模型仅取得12.24%的成功率,主要在GUI定位和操作知识方面遇到困难。使用OSWorld进行的综合分析为开发多模态通用智能体提供了宝贵见解,这是以往基准测试无法实现的。我们的代码、环境、基线模型和数据已在 https://os-world.github.io 公开。

英文摘要:

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑