arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.12481cs.AI
收藏

ToolCUA:迈向计算机使用代理的最优GUI工具路径协调

ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents

Tongyi Lab, Alibaba Group(通义实验室,阿里巴巴集团);Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Xuhao Hu, Xi Zhang, Haiyang Xu, Kyle Qiao, Jingyi Yang, Xuanjing Huang, Jing Shao, Ming Yan, Jieping Ye

AI总结:

本文提出ToolCUA,通过分阶段训练范式学习最优GUI-工具路径选择,利用交互式轨迹缩放管道和工具引导的GUI RFT提升关键切换点决策,实验显示其在OSWorld-MCP上准确率达46.85%,优于基线模型66%,证明了有效的GUI-工具协调。

摘要:

计算机使用代理(CUAs)可通过原子GUI操作如点击和输入,以及高级工具调用如基于API的文件操作进行行为,但这种混合操作空间常使它们在继续GUI操作还是切换到工具时不确定,导致执行路径不优。本文提出ToolCUA,一个端到端代理,通过分阶段训练范式学习最优GUI-工具路径选择。首先引入交互式GUI-工具轨迹缩放管道,利用丰富的静态GUI轨迹生成接地工具库,无需手动工程或真实工具轨迹收集即可生成多样化GUI-工具轨迹。然后进行工具引导的GUI RFT,结合预热SFT与单轮RL提升关键GUI-工具切换点决策。最后通过在线代理RL优化ToolCUA,在高保真的GUI-工具环境中,通过工具高效路径奖励鼓励适当使用工具和缩短执行路径。实验显示ToolCUA在OSWorld-MCP上准确率达46.85%,比基线模型提高约66%,建立新状态。同时比仅GUI设置提高3.9%,证明了有效的GUI-工具协调。结果进一步表明在混合操作空间训练是现实数字代理的有前景范式。开源地址:https://x-plug.github.io/ToolCUA/

英文摘要:

Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with GUI actions or switch to tools, leading to suboptimal execution paths. This difficulty stems from the scarcity of high-quality interleaved GUI-Tool trajectories, the cost and brittleness of collecting real tool trajectories, and the lack of trajectory-level supervision for GUI-Tool path selection. In this paper, we propose ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm. We first introduce an Interleaved GUI-Tool Trajectory Scaling Pipeline that repurposes abundant static GUI trajectories and synthesizes a grounded tool library, enabling diverse GUI-Tool trajectories without manual engineering or real tool-trajectory collection. We then perform Tool-Bootstrapped GUI RFT, combining warmup SFT with single-turn RL to improve decisions at critical GUI-Tool switching points. Finally, we optimize ToolCUA with Online Agentic RL in a high-fidelity GUI-Tool environment, guided by a Tool-Efficient Path Reward that encourages appropriate tool use and shorter execution paths. Experiments on OSWorld-MCP show that ToolCUA achieves 46.85% accuracy, a relative improvement of approximately 66% over the baseline, establishing a new state of the art among models of comparable scale. It also improves by 3.9% over GUI-only settings, demonstrating effective GUI-Tool orchestration. The results further suggest that training in a hybrid action space is a promising paradigm for real-world digital agents. Open-sourced here: https://x-plug.github.io/ToolCUA/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑