通过视觉反馈学习具有空间推理能力的 GUI 定位
Learning GUI Grounding with Spatial Reasoning from Visual Feedback
AI总结：
本文提出将 GUI 定位重构为交互式搜索任务，利用多步在线强化学习训练 GUI-Cursor 模型，通过光标视觉反馈提升空间推理能力，在 GUI 定位和代理任务上超越强基线。
摘要：
图形用户界面（GUI）定位通常被构建为坐标预测任务——给定自然语言指令，生成屏幕上用于点击和按键等操作的坐标。然而，最近的视觉语言模型（VLM）在处理高分辨率和复杂布局的 GUI 图像时，往往无法预测准确的数字坐标。为了解决这个问题，我们将 GUI 定位重构为交互式搜索任务，其中 VLM 生成动作以移动 GUI 中的光标来定位 UI 元素。在每一步，模型确定目标对象，评估光标与目标之间的空间关系，并根据移动历史将光标移近目标。在这个交互过程中，渲染的光标提供视觉反馈，帮助模型将其预测与相应的屏幕位置对齐。我们使用基于密集轨迹奖励函数的多步在线强化学习来训练我们的 GUI 定位模型 GUI-Cursor。实验结果表明，GUI-Cursor 在 GUI 定位和代理任务上超越了强基线，在相同基础模型下实现了更优性能，同时需要更少的训练数据。进一步分析表明，GUI-Cursor 学会在更困难的示例上自适应地执行更多步骤，并在分布外领域获得更好的空间推理能力。
英文摘要：
Graphical User Interface (GUI) grounding is commonly framed as a coordinate prediction task -- given a natural language instruction, generate on-screen coordinates for actions such as clicks and keystrokes. However, recent Vision Language Models (VLMs) often fail to predict accurate numeric coordinates when processing GUI images with high resolutions and complex layouts. To address this issue, we reframe GUI grounding as an interactive search task, where the VLM generates actions to move a cursor in the GUI to locate UI elements. At each step, the model determines the target object, evaluates the spatial relations between the cursor and the target, and moves the cursor closer to the target conditioned on the movement history. In this interactive process, the rendered cursor provides visual feedback to help the model align its predictions with the corresponding on-screen locations. We train our GUI grounding model, GUI-Cursor, using multi-step online reinforcement learning with a dense trajectory-based reward function. Experimental results demonstrate that GUI-Cursor surpasses strong baselines in GUI grounding and agentic tasks, achieving superior performance with the same base models while requiring less training data. Further analysis shows that GUI-Cursor learns to adaptively conduct more steps on more difficult examples, and it obtains better spatial reasoning capability on out-of-distribution domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
