arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.06391cs.CV
收藏

POINTS-GUI-G:GUI接地之旅

POINTS-GUI-G: GUI-Grounding Journey

Tencent(腾讯)

Zhongyin Zhao, Yuan Liu, Yikun Liu, Haicheng Wang, Le Tian, Xiao Zhou, Yangxiu You, Zilin Yu, Yang Yu, Jie Zhou

AI总结:

POINTS-GUI-G通过精细化数据工程、改进训练策略和强化学习提升GUI接地性能,实现多个基准测试的最优表现。

摘要:

视觉语言模型的快速发展催生了GUI代理,这些代理在自动化复杂任务(如在线购物和航班预订)中展现出巨大潜力,从而减轻了重复数字工作流程的负担。作为基础能力,GUI接地通常是端到端任务执行的前提条件。它使模型能够精确定位界面元素,如文本和图标,以执行点击和输入等准确操作。与以往工作不同,我们不是在已有强大空间意识的模型上进行微调(例如Qwen3-VL),而是从一个具有最小接地能力的基础模型开始,如POINTS-1.5。我们引入POINTS-GUI-G-8B,其在ScreenSpot-Pro上得分为59.9,在OSWorld-G上得分为66.0,在ScreenSpot-v2上得分为95.7,在UI-Vision上得分为49.9。我们的模型的成功由三个关键因素驱动:(1)精细化的数据工程,包括统一多样开源数据集格式以及复杂的增强、过滤和难度分级策略;(2)改进的训练策略,包括持续微调视觉编码器以提高感知精度,并保持训练和推理之间的分辨率一致性;(3)带有可验证奖励的强化学习(RL)。虽然RL传统上用于增强推理,但我们证明它在感知密集型的GUI接地任务中显著提高了精度。此外,GUI接地为RL提供了自然优势,因为奖励容易验证且高度准确。

英文摘要:

The rapid advancement of vision-language models has catalyzed the emergence of GUI agents, which hold immense potential for automating complex tasks, from online shopping to flight booking, thereby alleviating the burden of repetitive digital workflows. As a foundational capability, GUI grounding is typically established as a prerequisite for end-to-end task execution. It enables models to precisely locate interface elements, such as text and icons, to perform accurate operations like clicking and typing. Unlike prior works that fine-tune models already possessing strong spatial awareness (e.g., Qwen3-VL), we aim to master the full technical pipeline by starting from a base model with minimal grounding ability, such as POINTS-1.5. We introduce POINTS-GUI-G-8B, which achieves state-of-the-art performance with scores of 59.9 on ScreenSpot-Pro, 66.0 on OSWorld-G, 95.7 on ScreenSpot-v2, and 49.9 on UI-Vision. Our model's success is driven by three key factors: (1) Refined Data Engineering, involving the unification of diverse open-source datasets format alongside sophisticated strategies for augmentation, filtering, and difficulty grading; (2) Improved Training Strategies, including continuous fine-tuning of the vision encoder to enhance perceptual accuracy and maintaining resolution consistency between training and inference; and (3) Reinforcement Learning (RL) with Verifiable Rewards. While RL is traditionally used to bolster reasoning, we demonstrate that it significantly improves precision in the perception-intensive GUI grounding task. Furthermore, GUI grounding provides a natural advantage for RL, as rewards are easily verifiable and highly accurate.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑