arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.09362cs.CL
收藏

Falcon-UI:在遵循用户指令之前理解GUI

Falcon-UI: Understanding GUI Before Following User Instructions

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);College of Computer Science, Nankai University(南开大学计算机学院);Department of Automation, Tsinghua University(清华大学自动化系);Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)

Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, Xiangyang Ji

AI总结:

针对现有GUI智能体忽略上下文理解的问题,本文构建了无指令GUI导航数据集Insight-UI,提出先预训练学习GUI操作再指令微调的Falcon-UI模型,70亿参数规模即可达到720亿参数模型的性能水平。

摘要:

实现图形用户界面(GUI)智能体的类人交互,需要理解GUI上下文并遵循用户指令。然而,现有研究通常将这两个方面耦合在一起,且更侧重指令遵循能力,却忽略了理解GUI上下文的重要性。本文提出了一个无需指令的GUI导航数据集,名为Insight-UI Dataset,以增强模型对GUI环境的理解能力。Insight-UI Dataset由Common Crawl语料自动生成,覆盖31.2万个域名,模拟了iOS、Android、Windows和Linux等多种平台,且支持多种分辨率。尽管GUI交互会随上下文变化,但不同界面存在共通的内在模式,例如点击某个条目以查看其详情。这表明先独立学习GUI操作、再与指令调优进行联合优化是可行的。据此,我们开发了GUI智能体模型Falcon-UI:该模型首先在Insight-UI Dataset上进行预训练,随后在AITW、AITZ、Android Control和Mind2Web等Android与Web GUI数据集上进行微调。拥有70亿参数的Falcon-UI在AITZ基准上取得了与720亿参数的Qwen2VL相当的准确率,验证了GUI上下文理解能力与智能体性能的一致性。我们的代码和数据集将开源。

英文摘要:

Pursuing human-like interaction for Graphical User Interface (GUI) agents requires understanding the GUI context and following user instructions. However, existing works typically couple these two aspects and focus more on instruct-following abilities, while ignoring the importance of understanding the GUI context. In this paper, we introduce an instruction-free GUI navigation dataset, termed Insight-UI Dataset, to enhance model comprehension of GUI environments. Insight-UI Dataset is automatically generated from the Common Crawl corpus, simulating various platforms -- including iOS, Android, Windows, and Linux -- across multiple resolutions on 312K domains. Although GUI interactions vary by context, diverse interfaces share common internal patterns, such as clicking an item to view its details. It implies the feasibility of independent GUI operation learning, followed by joint optimization with instruction tuning. Thereby, we develop the GUI agent model Falcon-UI, which is initially pretrained on Insight-UI Dataset and subsequently fine-tuned on Android and Web GUI datasets, including AITW, AITZ, Android Control, and Mind2Web. With 7 billion parameters, Falcon-UI achieves accuracy comparable to the 72 billion-parameter Qwen2VL on AITZ, validating the alignment between GUI context comprehension and agent performance. Our code and dataset will be open-sourced.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑