WinClick：基于多模态大语言模型的GUI定位
WinClick: GUI Grounding with Multimodal Large Language Models
AI总结：
本文提出Windows平台视觉GUI智能体WinClick，通过截图检测可操作区域，结合GUI定位预训练和基于LLM的数据对齐方法解决GUI定位问题，并发布首个Windows GUI定位基准WinSpot，实验表明其显著优于现有基线。
摘要：
图形用户界面（GUI）任务对于自动化工作流（如软件测试、用户界面导航）至关重要。对用户而言，GUI是与计算机交互最直观的平台。先前工作指出了开发视觉GUI智能体的一个关键挑战：GUI定位——即根据指令准确定位屏幕元素的能力。然而，大多数现有GUI智能体在训练或推理中依赖结构化数据格式（如DOM或HTML文件），而这些格式并非在所有应用中都可访问，尤其是在Windows操作系统等通用桌面环境中。为解决这一问题，我们提出了WinClick，一个在Windows平台上开发的新型视觉GUI智能体。WinClick利用截图来检测可操作区域。为克服GUI定位的挑战，我们通过GUI定位预训练增强WinClick，并提出一种基于LLM的方法来对齐GUI定位数据。此外，我们引入了WinSpot，这是首个针对Windows上GUI定位的全面基准。我们的实验表明，结合GUI定位预训练的WinClick显著优于现有基线，为桌面环境中的GUI自动化提供了一种可扩展的解决方案。WinSpot已在https://github.com/zackhuiiiii/WinSpot公开可用。
英文摘要：
Graphical User Interface (GUI) tasks are vital for automating workflows such as software testing, user interface navigation. For users, the GUI is the most intuitive platform for interacting with a computer. Previous work identified a key challenge in developing visual GUI agents: GUI grounding - the ability to accurately locate screen elements based on instructions. However, most existing GUI agents rely on structured data formats like DOM or HTML files in training or inferencing, which are inaccessible across all applications, particular in a general desktop environments such as Windows OS. To address this, we introduce WinClick, a novel visual GUI agent developed in Windows platform. WinClick leverages screenshots to detect actionable regions. To overcome the challenge of GUI grounding, we enhance WinClick with GUI grounding pre-training and propose an LLM-based method for aligning GUI grounding data. Additionally, we introduce WinSpot, the first comprehensive benchmark for GUI grounding on Windows. Our experiments demonstrate that WinClick, combined with GUI grounding pre-training, significantly outperforms existing baselines, offering a scalable solution for GUI automation in desktop environments. WinSpot is publicly available at https://github.com/zackhuiiiii/WinSpot.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
