OS-ATLAS：面向通用GUI智能体的基础动作模型
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
AI总结：
针对开源VLM在GUI智能体任务中性能落后的问题，研究团队开发了基础GUI动作模型OS-Atlas，配套发布含超1300万元素的跨平台GUI定位语料库，在多平台基准上实现了性能突破。
摘要：
当前构建GUI智能体的研究工作高度依赖GPT-4o、GeminiProVision等性能强大的商用视觉语言模型（VLM）。由于开源VLM的性能与闭源模型存在显著差距，尤其是在GUI定位和分布外（OOD）场景下，从业者往往不愿使用开源VLM。为推动该领域的后续研究，我们开发了OS-Atlas——一款基础GUI动作模型，通过数据和建模两方面的创新，在GUI定位和分布外智能体任务中表现优异。我们投入大量工程精力开发了一套开源工具包，可合成Windows、Linux、MacOS、Android和网页等多平台的GUI定位数据。借助该工具包，我们发布了目前规模最大的开源跨平台GUI定位语料库，包含超过1300万个GUI元素。该数据集结合模型训练层面的创新，为OS-Atlas理解GUI截图并泛化到未见过的界面提供了坚实基础。通过在覆盖移动、桌面、网页三类平台的6个基准上开展全面评估，OS-Atlas的性能较此前的最优模型有显著提升。我们的评估还为持续提升和扩展开源VLM的智能体能力提供了宝贵见解。
英文摘要：
Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs due to their significant performance lag compared to their closed-source counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD) scenarios. To facilitate future research in this area, we developed OS-Atlas - a foundational GUI action model that excels at GUI grounding and OOD agentic tasks through innovations in both data and modeling. We have invested significant engineering effort in developing an open-source toolkit for synthesizing GUI grounding data across multiple platforms, including Windows, Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements. This dataset, combined with innovations in model training, provides a solid foundation for OS-Atlas to understand GUI screenshots and generalize to unseen interfaces. Through extensive evaluation across six benchmarks spanning three different platforms (mobile, desktop, and web), OS-Atlas demonstrates significant performance improvements over previous state-of-the-art models. Our evaluation also uncovers valuable insights into continuously improving and scaling the agentic capabilities of open-source VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
