GUI-Actor：面向 GUI 智能体的免坐标视觉定位
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
AI总结：
GUI-Actor 提出免坐标的 GUI 视觉定位方法，通过注意力动作头对齐 <ACTOR> token 与视觉块 token 并配合验证器选择动作区域，仅微调约 1 亿参数即可超越 UI-TARS-72B 等现有方法。
摘要：
构建由视觉语言模型（VLM）驱动的 GUI 智能体的主要挑战之一是视觉定位，即根据视觉内容和文本计划定位用于动作执行的合适屏幕区域。现有大多数工作将该任务形式化为基于文本的坐标生成任务。然而，这些方法存在若干局限：空间-语义对齐能力较弱，无法处理模糊的监督目标，以及屏幕坐标的密集特性与视觉Transformer等模型提取的块级粗粒度视觉特征之间存在不匹配。本文提出 GUI-Actor，一种基于 VLM 的免坐标 GUI 定位方法。其核心是引入一个基于注意力的动作头，该动作头学习将专用的 <ACTOR> token 与所有相关视觉块 token 对齐，使模型能够在单次前向传播中提出一个或多个动作区域。与此配套，我们进一步设计了一个定位验证器，用于评估并从候选动作区域中选择最合理的区域进行动作执行。大量实验表明，GUI-Actor 在多个 GUI 动作定位基准上优于先前最先进的方法，并且对未见过的屏幕分辨率和布局具有更好的泛化能力。值得注意的是，GUI-Actor-7B 在 ScreenSpot-Pro 上甚至超越了 UI-TARS-72B（38.1），以 Qwen2-VL 为骨干时得分 40.7，以 Qwen2.5-VL 为骨干时得分 44.6。此外，通过引入验证器，我们发现仅微调新引入的动作头（7B 模型约 1 亿参数）而保持 VLM 骨干冻结，就足以达到与先前最先进模型相当的性能，这表明 GUI-Actor 能够在不损害底层 VLM 通用能力的前提下赋予其有效的定位能力。
英文摘要：
One of the principal challenges in building VLM-powered GUI agents is visual grounding, i.e., localizing the appropriate screen region for action execution based on both the visual content and the textual plans. Most existing work formulates this as a text-based coordinate generation task. However, these approaches suffer from several limitations: weak spatial-semantic alignment, inability to handle ambiguous supervision targets, and a mismatch between the dense nature of screen coordinates and the coarse, patch-level granularity of visual features extracted by models like Vision Transformers. In this paper, we propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its core, GUI-Actor introduces an attention-based action head that learns to align a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the model to propose one or more action regions in a single forward pass. In line with this, we further design a grounding verifier to evaluate and select the most plausible action region from the candidates proposed for action execution. Extensive experiments show that GUI-Actor outperforms prior state-of-the-art methods on multiple GUI action grounding benchmarks, with improved generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7 with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by incorporating the verifier, we find that fine-tuning only the newly introduced action head (~100M parameters for 7B model) while keeping the VLM backbone frozen is sufficient to achieve performance comparable to previous state-of-the-art models, highlighting that GUI-Actor can endow the underlying VLM with effective grounding capabilities without compromising its general-purpose strengths.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
