发表机构
Zhejiang University; Peking University; Tsinghua University（浙江大学; 北京大学; 清华大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出HybridCUA框架，通过构建混合GUI与CLI轨迹数据集并采用两阶段训练，使计算机使用智能体学会选择性使用CLI，在OSWorld和WindowsAgentArena上分别提升14.8和4.0个百分点。
AI 中文摘要
计算机使用智能体（CUA）在完成数字任务方面已展现出强大的能力。然而，现有的CUA要么仅依赖图形用户界面（GUI）交互，这通常效率低下且容易出错，要么通过应用程序特定的API或工具来增强GUI交互，这需要大量的工程投入且难以跨应用扩展。我们认为，下一代CUA应将GUI交互与命令行界面（CLI）相结合，利用GUI的通用性和shell命令的高效性。然而，一个关键挑战是，当前模型在任务执行过程中不知道何时或如何使用CLI。为解决这一挑战，我们开发了一个数据构建流水线，生成三种类型的轨迹：仅GUI轨迹、仅CLI轨迹以及GUI与CLI交错的轨迹。该流水线产出了HybridCUA-8K数据集，包含5K条混合轨迹和3K个经过验证的RLVR任务。基于这些数据，我们提出了一个两阶段的训练框架：首先在构建的轨迹上进行监督微调，随后使用我们设计的CLI感知奖励进行强化学习，该奖励鼓励智能体有选择且可靠地使用CLI。实验表明，HybridCUA-9B在OSWorld上达到了53.6%的准确率，相比基础模型提升了14.8个百分点，并在WindowsAgentArena上将性能提升了4.0个百分点。这些结果证明了混合GUI与CLI范式对计算机使用智能体的有效性和跨平台泛化能力。
英文摘要
Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often inefficient and error prone, or augment GUI interactions with application specific APIs or tools, which require substantial engineering effort and are difficult to scale across applications. We argue that the next generation of CUAs should combine GUI interactions with the command line interface (CLI), leveraging the generality of the GUI and the efficiency of shell commands. A critical challenge, however, is that current models do not know when or how to use the CLI during task execution. To address this challenge, we develop a data construction pipeline that produces three types of trajectories: GUI only, CLI only, and interleaved GUI and CLI trajectories. This pipeline results in HybridCUA-8K, containing 5K hybrid trajectories and 3K verified RLVR tasks. Building on these data, we propose a training framework with two stages: supervised fine tuning on the constructed trajectories, followed by reinforcement learning with our CLI aware rewards that encourages agents to use the CLI selectively and reliably. Experiments show that HybridCUA-9B achieves 53.6% accuracy on OSWorld, improving over the base model by 14.8 percentage points, and improves performance on WindowsAgentArena by 4.0 percentage points. These results demonstrate the effectiveness and cross platform generalizability of the hybrid GUI and CLI paradigm for computer use agents.
CommentsProject Page: https://zjureal.com/HybridCUA/ Code: https://github.com/ZJU-REAL/HybridCUA
