R-VLM：面向精确GUI定位的区域感知视觉语言模型
R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
AI总结：
R-VLM通过放大区域提议和IoU感知目标函数，提升GUI元素定位精度，在多个基准上实现13%的定位精度提升和3.2-9.7%的导航任务精度改进。
摘要：
视觉智能体模型在图形用户界面（GUI）上自动化人类活动已成为一个前景广阔的研究方向，这得益于大型视觉语言模型（VLM）的进步。GUI自动化中的一个关键挑战是跨不同平台精确地定位界面元素。现有的纯视觉GUI智能体直接从大而杂乱的屏幕截图中定位元素，这要求它们处理大量无关信息，从而损害了其准确性。此外，这些方法通常采用基本的交叉熵损失来学习定位目标，与已建立的物体检测指标（如交并比（IoU））相比，无法有效捕捉定位质量。为解决这些问题，我们提出了R-VLM，一种新颖的GUI定位方法，利用放大区域提议来实现精确的元素定位。我们还提出了一种IoU感知的目标函数，促进模型向高IoU预测收敛。我们的方法弥合了VLM与传统物体检测技术之间的差距，在GUI定位基准ScreenSpot和AgentStudio上，跨多种GUI平台将最先进的定位精度提高了13%。此外，我们的R-VLM方法在AITW和Mind2Web基准上的GUI导航任务中，展示了3.2%至9.7%的绝对精度提升。
英文摘要：
Visual agent models for automating human activities on Graphical User Interfaces (GUIs) have emerged as a promising research direction, driven by advances in large Vision Language Models (VLMs). A critical challenge in GUI automation is the precise grounding of interface elements across diverse platforms. Existing vision-only GUI agents directly ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. In addition, these approaches typically employ basic cross-entropy loss for learning grounding objectives, which fails to effectively capture grounding quality compared to established object detection metrics like Intersection-over-Union (IoU). To address these issues, we introduce R-VLM, a novel GUI grounding approach that leverages zoomed-in region proposals for precise element localization. We also propose an IoU-aware objective function that facilitates model convergence toward high IoU predictions. Our approach bridges the gap between VLMs and conventional object detection techniques, improving the state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. In addition, our R-VLM approach shows 3.2-9.7% absolute accuracy improvements in GUI navigation tasks on the AITW and Mind2Web benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
