GUI-Eyes: 用于GUI代理视觉接地的工具增强感知
GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
AI总结：
GUI-Eyes通过分阶段策略推理和细粒度奖励反馈实现工具感知主动感知，显著提升GUI代理的接地准确率。
摘要：
最近在视觉语言模型（VLMs）和强化学习（RL）方面的进展推动了GUI自动化的发展。然而，大多数现有方法依赖于静态的一次性视觉输入和被动感知，缺乏能够自适应确定何时、是否以及如何观察界面的能力。我们提出了GUI-Eyes，一种用于GUI任务中主动视觉感知的强化学习框架。为了获得更具信息量的观察，智能体学会在两阶段推理过程中做出战略决策，决定是否以及如何调用视觉工具，如裁剪或缩放。为了支持这种行为，我们引入了一种渐进式感知策略，将决策过程分解为粗略探索和细粒度接地，由两级策略协调。此外，我们设计了一种空间连续的奖励函数，专门针对工具使用，它整合了位置接近性和区域重叠性，以提供密集的监督，并缓解GUI环境中常见的奖励稀疏性问题。在ScreenSpot-Pro基准测试中，GUI-Eyes-3B仅使用3k个标注样本就实现了44.8%的接地准确率，显著优于监督和基于RL的基线方法。这些结果表明，通过分阶段策略推理和细粒度奖励反馈实现的工具感知主动感知，对于构建稳健且数据高效的GUI代理至关重要。
英文摘要：
Recent advances in vision-language models (VLMs) and reinforcement learning (RL) have driven progress in GUI automation. However, most existing methods rely on static, one-shot visual inputs and passive perception, lacking the ability to adaptively determine when, whether, and how to observe the interface. We present GUI-Eyes, a reinforcement learning framework for active visual perception in GUI tasks. To acquire more informative observations, the agent learns to make strategic decisions on both whether and how to invoke visual tools, such as cropping or zooming, within a two-stage reasoning process. To support this behavior, we introduce a progressive perception strategy that decomposes decision-making into coarse exploration and fine-grained grounding, coordinated by a two-level policy. In addition, we design a spatially continuous reward function tailored to tool usage, which integrates both location proximity and region overlap to provide dense supervision and alleviate the reward sparsity common in GUI environments. On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3k labeled samples, significantly outperforming both supervised and RL-based baselines. These results highlight that tool-aware active perception, enabled by staged policy reasoning and fine-grained reward feedback, is critical for building robust and data-efficient GUI agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
