arXiv 2509.241332025-09-29cs.CVcs.CL

通用扫描器遇见专家定位器：用于鲁棒 GUI Grounding 的协同由粗到细框架

Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding

University of California, San Diego（加州大学圣地亚哥分校）；ByteDance（字节跳动）；University of California, Merced（加州大学默塞德分校）；University of Southern California（南加州大学）；University at Buffalo（布法罗大学）；The University of Queensland（昆士兰大学）

Zhecheng Li, Guoxian Song, Yiwei Wang, Zhen Xiong, Junsong Yuan, Yujun Cai

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 GMS 由粗到细框架，协同通用 VLM 扫描候选区域与专用 grounding 模型精确定位，在 ScreenSpot-Pro 上达到 35.7% 准确率，较独立模型提升约 10 倍。

摘要：

在图形用户界面（GUI）中对自然语言查询进行 grounding 是一项具有挑战性的任务，要求模型理解各种应用和系统中多样的 UI 元素，同时准确预测预期操作的空间坐标。为解决这一问题，我们提出 GMS：Generalist Scanner Meets Specialist Locator，这是一种协同的由粗到细框架，可有效提升 GUI grounding 性能。GMS 通过在框架内为通用视觉-语言模型（VLM）和小型、任务专用的 GUI grounding 模型分配不同角色，利用二者的互补优势。具体而言，通用 VLM 充当“Scanner（扫描器）”以识别潜在的感兴趣区域，而经过微调的 grounding 模型则充当“Locator（定位器）”，在这些区域内输出精确坐标。该设计灵感来自人类执行 GUI grounding 的方式：眼睛扫描界面，大脑专注于理解和定位。我们的整个框架包含五个阶段，并结合分层搜索与跨模态通信，以取得有前景的预测结果。在 ScreenSpot-Pro 数据集上的实验结果表明，尽管“Scanner”和“Locator”模型独立使用时分别仅达到 2.0% 和 3.7% 的准确率，但它们在 GMS 框架中的集成实现了 35.7% 的总体准确率，代表了 10 倍的提升。此外，GMS 在多种设置下显著优于其他强基线，展示了其在通用 GUI grounding 方面的鲁棒性和潜力。

英文摘要：

Grounding natural language queries in graphical user interfaces (GUIs) presents a challenging task that requires models to comprehend diverse UI elements across various applications and systems, while also accurately predicting the spatial coordinates for the intended operation. To tackle this problem, we propose GMS: Generalist Scanner Meets Specialist Locator, a synergistic coarse-to-fine framework that effectively improves GUI grounding performance. GMS leverages the complementary strengths of general vision-language models (VLMs) and small, task-specific GUI grounding models by assigning them distinct roles within the framework. Specifically, the general VLM acts as a 'Scanner' to identify potential regions of interest, while the fine-tuned grounding model serves as a 'Locator' that outputs precise coordinates within these regions. This design is inspired by how humans perform GUI grounding, where the eyes scan the interface and the brain focuses on interpretation and localization. Our whole framework consists of five stages and incorporates hierarchical search with cross-modal communication to achieve promising prediction results. Experimental results on the ScreenSpot-Pro dataset show that while the 'Scanner' and 'Locator' models achieve only $2.0\%$ and $3.7\%$ accuracy respectively when used independently, their integration within GMS framework yields an overall accuracy of $35.7\%$, representing a $10 \times$ improvement. Additionally, GMS significantly outperforms other strong baselines under various settings, demonstrating its robustness and potential for general-purpose GUI grounding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。