视觉大模型 / VLM

Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

Zhen Yang, Zi-Yi Dou, Di Feng, Forrest Huang, Anh Nguyen, Keen You, Omar Attia, Yuhao Yang, Michael Feng, Haotian Zhang, Ram Ramrakhya, Chao Jia, Jeffrey Nichols, Alexander Toshev, Yinfei Yang, Zhe Gan

机构 * Apple（苹果公司） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.LG

ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learning

Tomoyuki Kagaya, Subramanian Lakshmi, Anbang Ye, Thong Jing Yuan, Jayashree Karlekar, Sugiri Pranata, Natsuki Murakami, Akira Kinose, Yang You

机构 * Panasonic Connect Co., Ltd.（松下电器（株式会社）） ； Panasonic R&D Center（松下研发中心） ； National University of Singapore（新加坡国立大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI、cs.LG

DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models

Yuhan Hao, Zhengning Li, Lei Sun, Weilong Wang, Naixin Yi, Sheng Song, Caihong Qin, Mofan Zhou, Yifei Zhan, Xianpeng Lang

机构 * Li Auto Inc. 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Benchmark: https://huggingface.co/datasets/LiAuto-DriveAction/drive-action

ImaginationPolicy: Towards Generalizable, Precise and Reliable End-to-End Policy for Robotic Manipulation

Dekun Lu, Wei Gao, Kui Jia

机构 * Dekun Lu ∗ , Wei Gao ∗ and Kui Jia ∗（作者） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.AI、cs.LG

Comments First two authors contribute equally. Project page: https://sites.google.com/view/imaginationpolicy

Towards General Computer Control with Hierarchical Agents and Multi-Level Action Spaces

Zihan Dong, Xinyu Fan, Zixiang Tang, Yunqing Li

机构 * The University of Tokyo（东京大学） ； Lenovo US（联想美国） ； Advanced AI Technology Center（高级人工智能技术中心） 纠错

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.AI、cs.LG

Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance

Songsheng Wang, Rucheng Yu, Zhihang Yuan, Chao Yu, Feng Gao, Yu Wang, Derek F. Wong

机构 * NICS-EFC Lab, Department of Electronic Engineering, Tsinghua University（清华大学电子工程系NICS-EFC实验室） ； Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系） ； Infinigence AI ； Tsinghua University（清华大学） ； Zhongguancun Academy（中关村学院） 纠错

专题命中 GUI与屏幕智能体 ：visual language model(abstract)；分类 cs.AI、cs.LG

Comments 13 pages, 5 figures, Accepted by EMNLP 2025 (main conference)

TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning

Ziyuan Chen, Zhenghui Zhao, Zhangye Han, Miancan Liu, Xianhang Ye, Yiqing Li, Hongbo Min, Jinkui Ren, Xiantao Zhang, Guitao Cao

机构 * East China Normal University（东华大学） ； Alibaba Group（阿里巴巴集团） ； University of Electronic Science and Technology of China（电子科技大学） ； Wuhan University（武汉大学） ； Sun Yat-sen University（中山大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI、cs.LG

ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation

Zekai Zhang, Weiye Zhu, Hewei Pan, Xiangchen Wang, Rongtao Xu, Xing Sun, Feng Zheng

机构 * Southern University of Science and Technology, China（南方科技大学，中国） ； Spatialtemporal AI, China（时空AI，中国） ； MBZUAI, UAE（MBZUAI，阿联酋） ； Tencent Youtu Lab（腾讯优图实验室） 纠错

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV、cs.AI

SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control

Quanfeng Lu, Zhantao Ma, Shuai Zhong, Jin Wang, Dahai Yu, Michael K. Ng, Ping Luo

机构 * The University of Hong Kong（香港大学） ； Hong Kong Baptist University（香港 Baptist 大学） ； TCL Corporate Research (Hong Kong) Co., Ltd（TCL 香港研究院） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV、cs.AI

Comments 28 pages, 12 figures

WebSight: A Vision-First Architecture for Robust Web Agents

Tanvir Bhathal, Asanshay Gupta

机构 * Stanford University（斯坦福大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

Haiyang Guo, Fanhu Zeng, Fei Zhu, Jiayi Wang, Xukai Wang, Jingang Zhou, Hongbo Zhao, Wenzhuo Liu, Shijie Ma, Da-Han Wang, Xu-Yao Zhang, Cheng-Lin Liu

机构 * School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院） ； MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS） ； Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港科学与创新研究院人工智能与机器人中心） ； School of Computer and Information Engineering, Xiamen University of Technology（厦门理工大学计算机与信息工程学院） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV、cs.LG

Comments Preprint

Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks

Utkarsh Shandilya, Marsha Mariya Kappan, Sanyam Jain, Vijeta Sharma

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Robotic Visual Instruction

Yanbang Li, Ziyang Gong, Haoyang Li, Xiaoqi Huang, Haolan Kang, Guangping Bai, Xianzheng Ma

机构 * Imperial College London（伦敦帝国理工学院） ； Shanghai AI Laboratory（上海人工智能实验室） ； UC San Diego（加州大学圣地亚哥分校） ； VIVO ； South China University of Technology（华南理工大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Project website: https://robotic-visual-instruction.github.io/

Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models

Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, Chelsea Finn

机构 * Stanford University（斯坦福大学） ； University of California, Berkeley（加州大学伯克利分校） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI、cs.LG

Comments ICML 2025

OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis

Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, Zhiyong Wu

机构 * Shanghai AI Laboratory（上海人工智能实验室） ； The University of Hong Kong（香港大学） ； Johns Hopkins University（约翰霍普金斯大学） ； Shanghai Jiao Tong University（上海交通大学） ； University of Oxford（牛津大学） ； Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments ACL 2025 Camera Ready

Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation

Xinzge Gao, Chuanrui Hu, Bin Chen, Teng Li

机构 * Anhui University（安徽大学） ； Qihoo360（奇虎360） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning

Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Jie Xie, Wei Zhou, Wang Xu, Yuanheng Zhang, Zhou Su, Zhongwu Zhai, Xiaoming Liu, Yudong Mei, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, Maosong Sun

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI

Comments Updated results in Table 2 and Table 3; The project is available at https://github.com/OpenBMB/AgentCPM-GUI

Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments

Pranav Guruprasad, Yangyue Wang, Sudipta Chowdhury, Harshvardhan Sikka, Paul Pu Liang

机构 * Manifold Research ； MIT（麻省理工学院） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV、cs.LG

Comments 16 pages, 26 figures

DPO Learning with LLMs-Judge Signal for Computer Use Agents

Man Luo, David Cobbley, Xin Su, Shachar Rosenman, Vasudev Lal, Shao-Yen Tseng, Phillip Howard

机构 * Intel（英特尔） ； Thoughtworks 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M. Tamer Özsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, Perouz Taslakian, Spandana Gella, Sai Rajeswar

机构 * Mila - Quebec AI Institute（魁北克AI研究所） ； University of Waterloo（滑铁卢大学） ； National University of Singapore（新加坡国立大学） ； CIFAR AI Chair（CIFAR人工智能职位） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI

Comments This paper has been accepted to the 41st International Conference on Machine Learning (ICML 2025)

Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correction

Wenke Xia, Ruoxuan Feng, Dong Wang, Di Hu

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京耿丽人工智能学院） ； Shanghai AI Laboratory（上海人工智能实验室） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR2025

Breaking the Data Barrier -- Building GUI Agents Through Task Generalization

Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, Junxian He

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV、cs.AI

Comments 24 pages, 11 figures

A Taxonomy of Self-Handover

Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 8 figures, 1 table, Last updated on April 7th, 2025

SmolVLM: Redefining small and efficient multimodal models

Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, Thomas Wolf

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

STEVE: A Step Verification Pipeline for Computer-use Agent Training

Fanbin Lu, Zhisheng Zhong, Ziqin Wei, Shu Liu, Chi-Wing Fu, Jiaya Jia

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

MP-GUI: Modality Perception with MLLMs for GUI Understanding

Ziwei Wang, Weizhi Chen, Leyang Yang, Sheng Zhou, Shengchu Zhao, Hanbei Zhan, Jiongchao Jin, Liangcheng Li, Zirui Shao, Jiajun Bu

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments Paper accepted to CVPR 2025

MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulation

Pingrui Zhang, Xianqiang Gao, Yuhan Wu, Kehui Liu, Dong Wang, Zhigang Wang, Bin Zhao, Yan Ding, Xuelong Li

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI

EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices

Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, Mengwei Xu

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.AI、cs.LG

DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents

Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, Kun Shao

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments Paper and Appendix, 26 pages

Foundation Models for CPS-IoT: Opportunities and Challenges

Ozan Baris, Yizhuo Chen, Gaofeng Dong, Liying Han, Tomoyoshi Kimura, Pengrui Quan, Ruijie Wang, Tianchen Wang, Tarek Abdelzaher, Mario Bergés, Paul Pu Liang, Mani Srivastava

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG