arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.05941cs.CVcs.AIcs.CL
收藏

聚焦,点击:解锁和评估缩放在GUI定位中的潜力

Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding

Xi’an Jiaotong University(西安交通大学);Princeton University(普林斯顿大学);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);The University of Hong Kong(香港大学);Michigan State University(密歇根州立大学)

Zhiyuan Jiang, Shenghao Xie, Wenyi Li, Wenqiang Zu, Peihang Li, Jiahao Qiu, Siqi Pei, Lei Ma, Tiejun Huang, Mengdi Wang, Shilong Liu

AI总结:

本文提出ZoomClick方法,通过引入缩放先验知识提升GUI定位性能,实现动态空间聚焦与自适应上下文切换,并在多个基准上取得最佳结果。

摘要:

GUI代理构建中的定位是一项基本能力。尽管现有方法依赖于大规模边界框监督,但仍面临各种挑战,如跨平台泛化、复杂布局分析和细粒度元素定位。在本文中,我们研究缩放作为一种强大但未被充分探索的先验知识,提出了一个无需训练的方法ZoomClick。通过刻画缩放的四个关键属性(即预缩放、深度、缩小尺寸、最小裁剪尺寸),我们解锁了其在动态空间聚焦和自适应上下文切换方面的全部能力。实验表明,我们的方法显著提升了通用视觉-语言和专用GUI定位模型的性能,在几个主流基准上取得了最先进的结果;例如,UI-Venus-72B在ScreenSpot-Pro上达到了73.1%的成功率。此外,我们提出了GUIZoom-Bench,一个用于评估模型对缩放适应性的基准,旨在启发未来研究,以改进缩放在GUI定位任务中的进一步训练和测试时间扩展。

英文摘要:

Grounding is a fundamental capability for building graphical user interface (GUI) agents. Although existing approaches rely on large-scale bounding box supervision, they still face various challenges, such as cross-platform generalization, complex layout analysis, and fine-grained element localization. In this paper, we investigate zoom as a strong yet underexplored prior for GUI grounding, and propose a training-free method, ZoomClick. By characterizing four key properties of zoom (i.e., pre-zoom, depth, shrink size, minimal crop size), we unlock its full capabilities for dynamic spatial focusing and adaptive context switching. Experiments demonstrate that our method significantly boosts the performance of both general vision-language and specialized GUI grounding models, achieving state-of-the-art results on several mainstream benchmarks; for example, UI-Venus-72B attains a 73.1% success rate on ScreenSpot-Pro. Furthermore, we present GUIZoom-Bench, a benchmark for evaluating model adaptability to zoom, aiming to inspire future research on improving zoom for further training and test-time scaling in GUI grounding tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑