arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.24965cs.CVcs.AIcs.HC
收藏

ShowUI-$π$: 基于流的生成模型作为GUI灵巧手

ShowUI-$π$: Flow-based Generative Models as GUI Dexterous Hands

Show Lab, National University of Singapore(新加坡国立大学Show实验室)

Siyuan Hu, Kevin Qinghong Lin, Mike Zheng Shou

AI总结:

ShowUI-$π$ 是首个基于流的生成模型,通过统一动作、流式动作生成和拖动基准测试,实现了GUI灵巧手的高效拖动控制。

摘要:

构建能够实现灵巧操作的智能代理对于在机器人和数字环境中实现类人自动化至关重要。然而,现有的GUI代理依赖于离散的点击预测(x,y),这阻碍了需要连续实时感知和调整的自由形式闭环轨迹(例如拖动进度条)。在本工作中,我们开发了ShowUI-$π$,这是首个基于流的生成模型作为GUI灵巧手,其设计包括:(i)统一的离散-连续动作,将离散点击和连续拖动整合到共享模型中,使代理能够在多种交互模式中灵活适应;(ii)基于流的动作生成用于拖动建模,通过轻量级的动作专家从连续的视觉观察中预测增量光标调整,确保平滑且稳定的轨迹;(iii)拖动训练数据和基准测试,我们手动收集并合成了20,000条跨五个领域的拖动轨迹(例如PowerPoint、Adobe Premiere Pro),并引入ScreenDrag基准测试,该基准测试包含全面的在线和离线评估协议,用于评估GUI代理的拖动能力。我们的实验表明,专有GUI代理在ScreenDrag上仍面临挑战(例如,Operator得分为13.27,而最佳的Gemini-2.5-CUA达到22.18)。相比之下,ShowUI-$π$仅使用4.5亿参数就实现了26.98的得分,凸显了该任务的难度以及我们方法的有效性。我们希望这项工作能推动GUI代理向数字世界中类人灵巧控制迈进。代码可在https://github.com/showlab/showui-pi上获取。

英文摘要:

Building intelligent agents capable of dexterous manipulation is essential for achieving human-like automation in both robotics and digital environments. However, existing GUI agents rely on discrete click predictions (x,y), which prohibits free-form, closed-loop trajectories (e.g. dragging a progress bar) that require continuous, on-the-fly perception and adjustment. In this work, we develop ShowUI-$π$, the first flow-based generative model as GUI dexterous hand, featuring the following designs: (i) Unified Discrete-Continuous Actions, integrating discrete clicks and continuous drags within a shared model, enabling flexible adaptation across diverse interaction modes; (ii) Flow-based Action Generation for drag modeling, which predicts incremental cursor adjustments from continuous visual observations via a lightweight action expert, ensuring smooth and stable trajectories; (iii) Drag Training data and Benchmark, where we manually collect and synthesize 20K drag trajectories across five domains (e.g. PowerPoint, Adobe Premiere Pro), and introduce ScreenDrag, a benchmark with comprehensive online and offline evaluation protocols for assessing GUI agents' drag capabilities. Our experiments show that proprietary GUI agents still struggle on ScreenDrag (e.g. Operator scores 13.27, and the best Gemini-2.5-CUA reaches 22.18). In contrast, ShowUI-$π$ achieves 26.98 with only 450M parameters, underscoring both the difficulty of the task and the effectiveness of our approach. We hope this work advances GUI agents toward human-like dexterous control in digital world. The code is available at https://github.com/showlab/showui-pi.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑