GUI-AIMA: 对齐内在多模态注意力与上下文锚点以实现GUI定位
GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
AI总结：
GUI-AIMA通过多模态注意力对齐与上下文锚点，实现高效的GUI定位，无需坐标生成，提升数据效率和性能。
摘要：
图形用户界面（GUI）定位是计算机使用代理的关键能力，将自然语言指令映射到屏幕上的可操作区域。现有多模态大语言模型（MLLM）方法通常将GUI定位视为基于文本的坐标生成任务。然而，直接从视觉输入生成精确坐标具有挑战性且数据需求大。一种更直观的策略是首先识别与指令相关的视觉块，然后确定其中的精确点击位置。受最近观察到的通用MLLM内在定位能力的启发，我们提出了GUI-AIMA，一种基于注意力和无坐标的监督微调框架，用于高效的GUI定位。GUI-AIMA对齐MLLM的内在多模态注意力与块级定位信号。这些信号通过多头聚合在简化查询-视觉注意力矩阵上计算，以适应不同的用户指令。此外，其无坐标的特性可以轻松集成一个插件式的放大阶段。GUI-AIMA-3B仅用509k样本（约101k张截图）进行训练，展示了卓越的数据效率，并验证了轻量训练可以触发MLLM的内在定位能力。它在3B模型中实现了最先进的性能，平均准确率在ScreenSpot-Pro上为61.5%，在ScreenSpot-v2上为92.1%，在OSWorld-G上为68.1%，在MMBenchmark-GUI-L2上为79.1%，在UI-Vision上为60.0%。项目页面：https://github.com/sjz5202/GUI-AIMA
英文摘要：
Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen. Existing Multimodal Large Language Model (MLLM) approaches typically formulate GUI grounding as a text-based coordinate generation task. However, directly generating precise coordinates from visual inputs is challenging and often data-intensive. A more intuitive strategy is to first identify instruction-relevant visual patches and then determine the exact click location within them. Motivated by recent observations that general MLLMs exhibit native grounding ability embedded in their attention maps, we propose GUI-AIMA, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns the intrinsic multimodal attention of MLLMs with patch-wise grounding signals. These signals are calculated adaptively for diverse user instructions by multi-head aggregation on simplified query-visual attention matrices. Besides, its coordinate-free manner can easily integrate a plug-and-play zoom-in stage. GUI-AIMA-3B was trained with only 509k samples (around 101k screenshots), demonstrating exceptional data efficiency and verifying that light training can trigger the native grounding capability of MLLMs. It achieves state-of-the-art performance among 3B models, attaining an average accuracy of 61.5% on ScreenSpot-Pro, 92.1% on ScreenSpot-v2, 68.1% on OSWorld-G, 79.1% on MMBench-GUI-L2, and 60.0% on UI-Vision. Project page: https://github.com/sjz5202/GUI-AIMA .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
