面向纯视觉 GUI 智能体的 OmniParser
OmniParser for Pure Vision Based GUI Agent
AI总结：
本文提出 OmniParser，通过可交互图标检测与语义描述模型将界面截图解析为结构化元素，从而显著提升 GPT-4V 等纯视觉 GUI 智能体的动作定位与执行能力。
摘要：
近期大型视觉语言模型的成功展现了其在驱动作用于用户界面的智能体系统方面的巨大潜力。然而，我们认为，像 GPT-4V 这样强大的多模态模型，作为跨多个操作系统和不同应用的通用智能体，其能力被严重低估，原因在于缺乏一种稳健的屏幕解析技术，该技术需要能够：1）可靠地识别用户界面中可交互的图标，以及 2）理解截图中各种元素的语义，并准确地将预期动作与屏幕上相应的区域关联起来。为填补这些空白，我们提出了 OmniParser，这是一种将用户界面截图解析为结构化元素的综合方法，它显著增强了 GPT-4V 生成能够准确定位到界面相应区域的动作的能力。我们首先利用热门网页构建了一个可交互图标检测数据集，并构建了一个图标描述数据集。这些数据集被用于微调专门的模型：一个检测模型用于解析屏幕上可交互的区域，一个描述模型用于提取所检测元素的功能语义。OmniParser 显著提升了 GPT-4V 在 ScreenSpot 基准上的性能。在 Mind2Web 和 AITW 基准上，仅使用截图输入的 OmniParser 也优于需要截图之外额外信息的 GPT-4V 基线。
英文摘要：
The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce \textsc{OmniParser}, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements. \textsc{OmniParser} significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, \textsc{OmniParser} with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
