arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.11784cs.LGcs.AIcs.CLcs.CV
收藏

ClawGUI:一个用于训练、评估和部署GUI代理的统一框架

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

Zhejiang University(浙江大学)

Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

AI总结:

ClawGUI通过统一框架解决GUI代理训练、评估和部署中的三大缺口,提供标准化评估流程和跨平台部署能力,实现移动端GUI任务的成功率提升。

摘要:

GUI代理通过视觉界面驱动应用程序,而非程序接口,通过点击、滑动和按键与任意软件交互,覆盖CLI代理无法触及的长尾应用。然而,该领域进展受限于缺乏统一的全栈基础设施:在线RL训练受环境不稳定和封闭管道影响,评估协议在不同工作中悄然漂移,训练的代理 rarely 达到真实用户。我们提出ClawGUI,一个开源框架,通过单一框架解决这三个缺口。ClawGUI-RL提供首个开源GUI代理RL基础设施,支持并行虚拟环境和真实物理设备,整合GiGPO与过程奖励模型实现密集步骤级监督。ClawGUI-Eval在6个基准和11+模型上实现标准化评估流程,达到95.8%的再生产率。ClawGUI-Agent通过12+聊天平台将训练的代理部署到Android、HarmonyOS和iOS,实现混合CLI-GUI控制和持久个性化记忆。在该流程中训练的ClawGUI-2B在MobileWorld GUI-Only任务中达到17.1%的成功率,优于相同规模的MAI-UI-2B基线6.0%。

英文摘要:

GUI agents drive applications through their visual interfaces instead of programmatic APIs, interacting with arbitrary software via taps, swipes, and keystrokes, reaching a long tail of applications that CLI-based agents cannot. Yet progress in this area is bottlenecked less by modeling capacity than by the absence of a coherent full-stack infrastructure: online RL training suffers from environment instability and closed pipelines, evaluation protocols drift silently across works, and trained agents rarely reach real users on real devices. We present \textbf{ClawGUI}, an open-source framework addressing these three gaps within a single harness. \textbf{ClawGUI-RL} provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices, integrating GiGPO with a Process Reward Model for dense step-level supervision. \textbf{ClawGUI-Eval} enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models, achieving 95.8\% reproduction against official baselines. \textbf{ClawGUI-Agent} brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory. Trained end to end within this pipeline, \textbf{ClawGUI-2B} achieves 17.1\% Success Rate on MobileWorld GUI-Only, outperforming the same-scale MAI-UI-2B baseline by 6.0\%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑