GUICourse:从通用视觉语言模型到多功能GUI智能体
GUICourse: From General Vision Language Models to Versatile GUI Agents
AI总结:
本文提出GUICourse数据集套件,通过引入GUIEnv增强VLMs的OCR和grounding能力,并结合GUIAct和GUIChat丰富GUI组件与交互知识,成功训练出在常见单步及多步GUI任务上优于基线VLMs的视觉GUI智能体。
摘要:
利用图形用户界面(GUI)进行人机交互对于访问广泛的数字工具至关重要。视觉语言模型(VLMs)的最新进展凸显了开发多功能智能体以帮助人类完成GUI导航任务的巨大潜力。然而,当前的VLMs在基础能力(OCR和grounding)以及GUI知识(GUI元素的功能和控制方法)方面面临挑战,阻碍了它们成为实用的GUI智能体。为了解决这些挑战,我们贡献了GUICourse,一套用于从通用VLMs训练基于视觉的GUI智能体的数据集。首先,我们引入了GUIEnv数据集以增强VLMs的OCR和grounding能力。然后,我们引入了GUIAct和GUIChat数据集以丰富它们对GUI组件和交互的知识。实验表明,我们的GUI智能体在常见GUI任务上的表现优于其基线VLMs。即使是小规模的GUI智能体(具有3.1B参数)也能在单步和多步GUI任务上良好运行。最后,我们通过消融研究分析了该智能体训练阶段的不同变体。我们的源代码和数据集已在 https://github.com/yiye3/GUICourse 发布。
英文摘要:
Utilizing Graphic User Interface (GUI) for human-computer interaction is essential for accessing a wide range of digital tools. Recent advancements in Vision Language Models (VLMs) highlight the compelling potential to develop versatile agents to help humans finish GUI navigation tasks. However, current VLMs are challenged in terms of fundamental abilities (OCR and grounding) and GUI knowledge (the functions and control methods of GUI elements), preventing them from becoming practical GUI agents. To solve these challenges, we contribute GUICourse, a suite of datasets to train visual-based GUI agents from general VLMs. First, we introduce the GUIEnv dataset to strengthen the OCR and grounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChat datasets to enrich their knowledge of GUI components and interactions. Experiments demonstrate that our GUI agents have better performance on common GUI tasks than their baseline VLMs. Even the small-size GUI agent (with 3.1B parameters) can still work well on single-step and multi-step GUI tasks. Finally, we analyze the different varieties in the training stage of this agent by ablation study. Our source codes and datasets are released at https://github.com/yiye3/GUICourse.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。