arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.08047cs.AIcs.MA
收藏

WorldGUI: 一个从任意起点进行桌面GUI自动化的交互式基准测试

WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point

Show Lab, National University of Singapore(新加坡国立大学Show实验室)

Henry Hengyuan Zhao, Kaiming Yang, Wendi Yu, Difei Gao, Mike Zheng Shou

AI总结:

提出WorldGUI基准测试,覆盖10个桌面和Web应用,在多种系统构建的初始状态下评估GUI代理的规划鲁棒性,并引入WorldGUI-Agent框架通过三阶段批评提升动态环境下的可靠性。

摘要:

近期GUI代理的进展显著提升了视觉定位能力,但稳健的规划仍然具有挑战性,特别是当环境偏离规范初始状态时。在实际应用中,用户通常在工作流程中请求帮助,此时软件可能已部分配置,步骤可能以不同顺序执行,或者界面可能与默认设置不同。这种任务状态变异性普遍存在,但在现有GUI基准测试中评估不足。为解决这一问题,我们引入了WorldGUI,一个涵盖十种广泛使用的桌面和Web应用的基准测试,其任务在多样化、系统构建的初始状态下实例化。这些变化捕捉了真实的人机交互场景,并能够诊断评估代理恢复、调整计划以及处理非默认上下文的能力。我们进一步提出了WorldGUI-Agent,一个简单且与模型无关的框架,围绕三个批评阶段组织规划和执行,提高了动态环境中的可靠性。实验表明,最先进的GUI代理在非默认初始条件下表现出显著的性能下降,揭示了有限的鲁棒性和脆弱的规划行为。我们的基准测试和框架为开发更适应性和可靠的GUI代理奠定了基础。代码和数据可在https://github.com/showlab/WorldGUI获取。

英文摘要:

Recent progress in GUI agents has substantially improved visual grounding, yet robust planning remains challenging, particularly when the environment deviates from a canonical initial state. In real applications, users often invoke assistance mid-workflow, where software may be partially configured, steps may have been executed in different orders, or the interface may differ from its default setup. Such task-state variability is pervasive but insufficiently evaluated in existing GUI benchmarks. To address this gap, we introduce WorldGUI, a benchmark covering ten widely used desktop and web applications with tasks instantiated under diverse, systematically constructed initial states. These variations capture realistic human-computer interaction settings and enable diagnostic evaluation of an agent's ability to recover, adapt plans, and handle non-default contexts. We further present WorldGUI-Agent, a simple and model-agnostic framework that organizes planning and execution around three critique stages, improving reliability in dynamic environments. Experiments demonstrate that state-of-the-art GUI agents exhibit substantial performance degradation under non-default initial conditions, revealing limited robustness and fragile planning behaviors. Our benchmark and framework provide a foundation for developing more adaptable and reliable GUI agents. The code and data are available at https://github.com/showlab/WorldGUI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑