AdaZoom-GUI: 基于自适应缩放的GUI定位与指令细化
AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
AI总结：
本文提出AdaZoom-GUI框架，通过指令细化模块和条件缩放策略提升GUI定位精度与指令理解，构建高质量数据集并采用GRPO训练模型，实现在高分辨率GUI理解中的最佳性能。
摘要：
GUI定位是视觉-语言模型（VLMs）的关键能力，通过自然语言指令定位目标元素实现自动交互。然而，由于高分辨率图像、小尺寸UI元素和模糊用户指令，GUI截图定位仍具挑战性。本文提出AdaZoom-GUI，一种基于自适应缩放的GUI定位框架，提升定位精度和指令理解。我们的方法引入指令细化模块，将自然语言指令转换为明确详细描述，使定位模型聚焦于精确元素定位。此外，我们设计了条件缩放策略，对预测的小元素进行二次推理，提升定位精度同时避免不必要的计算和上下文丢失。为支持该框架，我们构建了高质量的GUI定位数据集，并使用组相对策略优化（GRPO）训练定位模型，使其能够预测点击坐标和元素边界框。在公开基准测试中，我们的方法在参数规模相当或更大的模型中实现了最先进的性能，突显其在高分辨率GUI理解和实际GUI代理部署中的有效性。
英文摘要：
GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
