OS-Helix：在原生计算机使用智能体中统一GUI交互与代码执行
OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents
AI总结：
OS-Helix是端到端训练的多模态计算机使用智能体，通过应用-工件组合等方法解决GUI与代码协调问题，在OSWorld-Verified上较Qwen3.5基线实现显著性能提升，且可跨平台迁移。
摘要：
真实的桌面自动化既非纯视觉也非纯程序化，智能体必须理解面向用户的界面，同时高效操作文件、脚本和系统状态。然而，原生GUI与代码的协调无法通过提示级工具组合实现。在初步实验中，单纯添加代码工具会降低GUI智能体的性能，这表明GUI与代码的路由必须通过高质量混合数据的训练来内化。我们提出OS-Helix，一种端到端训练的多模态计算机使用智能体，在单一动作空间中统一GUI动作与代码执行。为构建所需的监督信号，我们引入应用-工件组合，将桌面应用与真实文件工件配对以实例化可执行的Ubuntu状态，以及一种解耦的指令/评估器生成流水线，生成基于环境的任务，无论解决方案模态如何，其最终状态均可验证。我们通过对过滤后的混合数据进行SFT来冷启动策略，并在实时桌面环境中通过在线RL进一步改进。对于RL，我们提出Step-balanced GSPO，其按动作步骤而非原始助手令牌聚合重要性权重，以防止长代码动作主导异构动作的优化。在OSWorld-Verified上，OS-Helix-27B和OS-Helix-35B-A3B相较于其匹配的Qwen3.5混合基线分别实现了35.0%和33.7%的相对提升，而OS-Helix-27B-RL进一步提升42.9%，达到66.44%的成功率。消融实验提供了与学习到的GUI/代码协调、可扩展可验证监督以及动作步骤平衡优化的贡献一致的证据，同时展示了在WindowsAgentArena上的跨平台迁移能力。我们的项目主页位于此https URL。
英文摘要：
Real desktop automation is neither purely visual nor purely programmatic. Agents must understand user-facing interfaces while efficiently manipulating files, scripts, and system state. However, native GUI/code coordination cannot be obtained by prompt-level tool composition. In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data. We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space. To construct the required supervision, we introduce application-artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality. We cold-start the policy with SFT on filtered hybrid data and further improve it through online RL in live desktop environments. For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization. On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B achieve relative improvements of 35.0% and 33.7% over their matched Qwen3.5 hybrid baselines, while OS-Helix-27B-RL further improves by 42.9% to reach 66.44% success. Ablations provide evidence consistent with contributions from learned GUI/code coordination, scalable verifiable supervision, and action step balanced optimization, while demonstrating cross-platform transfer on WindowsAgentArena. Our project homepage is at this https URL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
