通过显式位置到坐标映射改进 GUI 定位
Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
AI总结：
针对 VLM 在未知高分辨率 GUI 中坐标映射失效的问题，提出 RULER 显式坐标标记与 I-MRoPE 等宽高空间编码，在多个 ScreenSpot 基准上持续提升定位准确率。
摘要：
GUI 定位是将自然语言指令映射到像素坐标的任务，对自主智能体至关重要，但对当前 VLM 仍然困难。核心瓶颈在于可靠的 patch 到像素映射；在外推到训练期间未见过的高分辨率显示器时，这种映射会失效。现有方法直接从视觉特征生成作为文本 token 的坐标，迫使模型隐式推断复杂的位置到像素映射；因此，在新分辨率上准确率下降且失败增多。我们通过两项互补创新解决该问题。首先，RULER token 充当显式坐标标记，使模型能够像参照地图上的网格线一样参照位置并进行调整，而不是从头生成坐标。其次，Interleaved MRoPE（I-MRoPE）通过确保宽度和高度维度得到平等表示来改进空间编码，解决标准位置方案的不对称性。在 ScreenSpot、ScreenSpot-V2 和 ScreenSpot-Pro 上的实验表明，定位准确率持续提升，其中在高分辨率界面上的改进最大。通过提供显式空间引导而非依赖隐式学习，我们的方法能够在不同分辨率和平台上实现更可靠的 GUI 自动化。
英文摘要：
GUI grounding, the task of mapping natural-language instructions to pixel coordinates, is crucial for autonomous agents, yet remains difficult for current VLMs. The core bottleneck is reliable patch-to-pixel mapping, which breaks when extrapolating to high-resolution displays unseen during training. Current approaches generate coordinates as text tokens directly from visual features, forcing the model to infer complex position-to-pixel mappings implicitly; as a result, accuracy degrades and failures proliferate on new resolutions. We address this with two complementary innovations. First, RULER tokens serve as explicit coordinate markers, letting the model reference positions similar to gridlines on a map and adjust rather than generate coordinates from scratch. Second, Interleaved MRoPE (I-MRoPE) improves spatial encoding by ensuring that width and height dimensions are represented equally, addressing the asymmetry of standard positional schemes. Experiments on ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro show consistent gains in grounding accuracy, with the largest improvements on high-resolution interfaces. By providing explicit spatial guidance rather than relying on implicit learning, our approach enables more reliable GUI automation across diverse resolutions and platforms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
