Falcon-UI: Understanding GUI Before Following User Instructions
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);College of Computer Science, Nankai University(南开大学计算机学院);Department of Automation, Tsinghua University(清华大学自动化系);Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, Xiangyang Ji
Pursuing human-like interaction for Graphical User Interface (GUI) agents requires understanding the GUI context and following user instructions. However, existing works typically couple these two aspects and focus more on instruct-following abilities, while ignoring the importance of understanding the GUI context. In this paper, we introduce an instruction-free GUI navigation dataset, termed Insight-UI Dataset, to enhance model comprehension of GUI environments. Insight-UI Dataset is automatically generated from the Common Crawl corpus, simulating various platforms -- including iOS, Android, Windows, and Linux -- across multiple resolutions on 312K domains. Although GUI interactions vary by context, diverse interfaces share common internal patterns, such as clicking an item to view its details. It implies the feasibility of independent GUI operation learning, followed by joint optimization with instruction tuning. Thereby, we develop the GUI agent model Falcon-UI, which is initially pretrained on Insight-UI Dataset and subsequently fine-tuned on Android and Web GUI datasets, including AITW, AITZ, Android Control, and Mind2Web. With 7 billion parameters, Falcon-UI achieves accuracy comparable to the 72 billion-parameter Qwen2VL on AITZ, validating the alignment between GUI context comprehension and agent performance. Our code and dataset will be open-sourced.
Reclaiming Low-Quality Instruction Data with Rule-Guided LLM Reconstruction
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of California, Santa Cruz(加州大学圣克鲁兹分校);Hong Kong Baptist University(香港 Baptist 大学)
ConsistentChat: Building Skeleton-Guided Consistent Multi-Turn Dialogues for Large Language Models from Scratch
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China(中国科学院软件研究所中文信息处理实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学)
Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation
State Key Lab of CAD&CG(CAD与CG国家重点实验室);School of Computing(计算机学院);National University of Singapore(新加坡国立大学);School of Intelligence Science and Engineering(智能科学与工程学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responses
Zhejiang University of Science and Technology(浙江科技大学);Fudan University(复旦大学);Zhejiang University(浙江大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学)
LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
University of Chinese Academy of Sciences(中国科学院大学);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所认知与决策智能复杂系统重点实验室)
TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc(百度公司)