Ferret-UI 2：掌握跨平台的通用用户界面理解
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
AI总结：
Ferret-UI 2是一种多模态大语言模型，通过多平台支持、自适应高分辨率感知和GPT-4o生成训练数据，显著提升跨平台UI理解与交互性能。
摘要：
构建一个通用的用户界面（UI）理解模型面临诸多基础性挑战，例如平台多样性、分辨率差异和数据限制。在本文中，我们介绍了 Ferret-UI 2，一种多模态大语言模型（MLLM），旨在实现跨多种平台的通用UI理解，包括iPhone、Android、iPad、Webpage和AppleTV。在Ferret-UI的基础上，Ferret-UI 2引入了三项关键创新：支持多种平台类型、通过自适应缩放实现高分辨率感知，以及由GPT-4o结合set-of-mark视觉提示生成的高级任务训练数据。这些进展使Ferret-UI 2能够执行复杂的、以用户为中心的交互，使其在日益多样化的平台生态系统中具有高度的通用性和适应性。在指代、定位、以用户为中心的高级任务（包括9个子任务×5个平台）、GUIDE下一步动作预测数据集以及GUI-World多平台基准上的广泛实证实验表明，Ferret-UI 2显著优于Ferret-UI，并展现出强大的跨平台迁移能力。
英文摘要：
Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
