ShowUI：面向GUI视觉智能体的单一视觉-语言-动作模型
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
AI总结：
针对现有GUI助手依赖文本元信息、难以像人类感知UI视觉的局限，提出视觉-语言-动作模型ShowUI，通过UI引导Token选择、交错式流机制及高质量数据集构建，实现零样本截图定位75.1%准确率，减少33%冗余Token并提速1.4倍，推进GUI视觉智能体发展。
摘要：
构建图形用户界面（GUI）助手对提升人类工作流效率具有重要前景。多数智能体基于语言，依赖含丰富文本元信息的闭源API（如HTML或无障碍树），但在像人类一样感知UI视觉方面存在局限，凸显对GUI视觉智能体的需求。本工作在数字世界中开发了视觉-语言-动作模型ShowUI，具有以下创新：（i）UI引导的视觉Token选择，通过将截图构建为UI连接图，自适应识别冗余关系，并作为自注意力块中Token选择的标准，以降低计算成本；（ii）交错式视觉-语言-动作流，灵活统一GUI任务中的多样化需求，在导航中有效管理视觉-动作历史，或为每张截图配对多轮查询-动作序列以提升训练效率；（iii）小规模高质量GUI指令跟随数据集，通过精心的数据整理和重采样策略解决显著的数据类型不平衡问题。借助上述组件，使用256K数据的轻量级2B模型ShowUI在零样本截图定位中达到75.1%的高准确率。其UI引导的Token选择进一步在训练中减少33%的冗余视觉Token，并将性能提速1.4倍。在Web端Mind2Web、移动端AITW和在线MiniWob环境中的导航实验，进一步凸显了我们的模型在推进GUI视觉智能体方面的有效性和潜力。模型可在https://github.com/showlab/ShowUI获取。
英文摘要：
Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. While most agents are language-based, relying on closed-source API with text-rich meta-information (e.g., HTML or accessibility tree), they show limitations in perceiving UI visuals as humans do, highlighting the need for GUI visual agents. In this work, we develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations: (i) UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks; (ii) Interleaved Vision-Language-Action Streaming that flexibly unifies diverse needs within GUI tasks, enabling effective management of visual-action history in navigation or pairing multi-turn query-action sequences per screenshot to enhance training efficiency; (iii) Small-scale High-quality GUI Instruction-following Datasets by careful data curation and employing a resampling strategy to address significant data type imbalances. With above components, ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection further reduces 33% of redundant visual tokens during training and speeds up the performance by 1.4x. Navigation experiments across web Mind2Web, mobile AITW, and online MiniWob environments further underscore the effectiveness and potential of our model in advancing GUI visual agents. The models are available at https://github.com/showlab/ShowUI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
