Ferret-UI：基于多模态LLM的 grounded 移动UI理解
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
AI总结：
本文提出Ferret-UI多模态大模型，通过任意分辨率子图编码、区域标注指令数据与高级推理任务训练，提升移动UI指称、定位和推理能力，并在综合基准上超越多数开源模型及GPT-4V。
摘要：
近来，多模态大语言模型（MLLM）取得了显著进展，但这些通用领域的MLLM在有效理解用户界面（UI）屏幕并与之交互方面往往存在不足。本文提出Ferret-UI，这是一种专为增强移动UI屏幕理解而定制的新型MLLM，具备指称、定位和推理能力。鉴于UI屏幕通常呈现更狭长的宽高比，并且比自然图像包含更小的感兴趣对象（如图标、文本），我们在Ferret之上引入“任意分辨率”以放大细节，并利用增强的视觉特征。具体而言，每个屏幕根据原始宽高比被划分为2个子图像（即竖屏采用水平划分，横屏采用垂直划分）。两个子图像在送入LLM之前分别进行编码。我们从广泛的基础UI任务中精心收集训练样本，如图标识别、查找文本和组件列举。这些样本被格式化为带有区域标注的指令跟随数据，以促进精确的指称和定位。为增强模型的推理能力，我们进一步构建了一个面向高级任务的数据集，包括详细描述、感知/交互对话以及功能推断。在经过筛选的数据集上训练后，Ferret-UI表现出对UI屏幕的出色理解以及执行开放式指令的能力。在模型评估方面，我们建立了一个涵盖上述所有任务的综合基准。Ferret-UI不仅优于大多数开源UI MLLM，还在所有基础UI任务上超过GPT-4V。
英文摘要：
Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate "any resolution" on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model's reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
