GoClick：轻量级元素接地模型用于自主GUI交互
GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
AI总结：
本文提出GoClick，一种轻量级GUI元素接地模型，通过改进架构和数据精炼流程，在保持高精度的同时降低参数规模，适用于资源受限设备的低延迟GUI交互任务。
摘要：
图形用户界面（GUI）元素接地（基于自然语言指令精确定位截图中的元素）是与GUI交互代理的基础。在资源受限设备（如手机）上直接部署这一能力对需要低延迟的GUI代理越来越关键。然而，当前的视觉接地方法通常使用大型视觉语言模型（VLM）（超过25亿参数），这在设备端执行时由于内存和计算限制而不切实际。为了解决这个问题，本文引入了GoClick，一种仅含23亿参数的轻量级GUI元素接地VLM，实现了出色的视觉接地精度，甚至与显著更大的模型相当。简单地缩小现有解码器-only VLMs是一种设计轻量模型的直接方法，但我们的实验表明这种方法会产生次优结果。相反，我们选择了一种编码器-解码器架构，该架构在GUI接地任务中在小参数规模下优于解码器-only替代方案。此外，小VLM的有限容量促使我们开发了一种渐进式数据精炼流程，利用任务类型过滤和数据比例调整从1080万原始数据集中提取出高质量的380万样本核心集。使用这个核心集训练GoClick带来了显著的接地精度提升。我们的实验表明，GoClick在多个GUI元素接地基准测试中表现优异，同时保持了小的规模和高的推理速度。GoClick在集成到设备-云协作框架中时，也增强了GUI代理的性能，其中GoClick帮助基于云的任务规划器实现精确的元素定位并取得更高的成功率。我们希望我们的方法能为GUI代理社区提供有意义的探索。
英文摘要：
Graphical User Interface (GUI) element grounding (precisely locating elements on screenshots based on natural language instructions) is fundamental for agents interacting with GUIs. Deploying this capability directly on resource-constrained devices like mobile phones is increasingly critical for GUI agents requiring low latency. However, this goal faces a significant challenge, as current visual grounding methods typically employ large vision-language model (VLM) (more than 2.5B parameters), making them impractical for on-device execution due to memory and computational constraints. To address this, this paper introduces GoClick, a lightweight GUI element grounding VLM with only 230M parameters that achieves excellent visual grounding accuracy, even on par with significantly larger models. Simply downsizing existing decoder-only VLMs is a straightforward way to design a lightweight model, but our experiments reveal that this approach yields suboptimal results. Instead, we select an encoder-decoder architecture, which outperforms decoder-only alternatives at small parameter scales for GUI grounding tasks. Additionally, the limited capacity of small VLMs encourages us to develop a Progressive Data Refinement pipeline that utilizes task type filtering and data ratio adjustment to extract a high-quality 3.8M-sample core set from a 10.8M raw dataset. Training GoClick using this core set brings notable grounding accuracy gains. Our experiments show that GoClick excels on multiple GUI element grounding benchmarks while maintaining a small size and high inference speed. GoClick also enhances GUI agent performance when integrated into a device-cloud collaboration framework, where GoClick helps cloud-based task planners perform precise element localization and achieve higher success rates. We hope our method serves as a meaningful exploration within the GUI agent community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
