视觉大模型 / VLM

GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devices

Xudong Lu, Yinghao Chen, Renshou Wu, Haohao Gao, Xi Chen, Xue Yang, Xiangyu Zhao, Aojun Zhou, Fangyuan Li, Yafei Wen, Xiaoxin Chen, Shuai Ren, Hongsheng Li

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments 14 pages

CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planning

Yuqi Zhou, Shuai Wang, Sunhao Dai, Qinglin Jia, Zhaocheng Du, Zhenhua Dong, Jun Xu

专题命中 GUI与屏幕智能体 ：VLM(abstract)；visual language model(abstract)；分类 cs.AI

Lightweight Neural App Control

Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, Kun Shao

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments ICLR 2025 (spotlight)

SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living

Arkaprava Sinha, Dominick Reilly, Francois Bremond, Pu Wang, Srijan Das

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；vision language model(abstract)；分类 cs.CV

On AI-Inspired UI-Design

Jialiang Wei, Anne-Lise Courbis, Thomas Lambolais, Gérard Dray, Walid Maalej

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos

Reza Ghoddoosian, Nakul Agarwal, Isht Dwivedi, Behzad Darisuh

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Accepted at WACV 2025

OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, Yu Qiao

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)；分类 cs.CV

OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving

Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, Wenchao Ding

专题命中 GUI与屏幕智能体 ：visual question answering(abstract)；MLLM(abstract)；分类 cs.CV

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, Tao Yu

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；VLM(abstract)；分类 cs.AI

Comments 34 pages, 14 figures, 10 tables

Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs

Hao-Tien Lewis Chiang, Zhuo Xu, Zipeng Fu, Mithun George Jacob, Tingnan Zhang, Tsang-Wei Edward Lee, Wenhao Yu, Connor Schenck, David Rendleman, Dhruv Shah, Fei Xia, Jasmine Hsu, Jonathan Hoech, Pete Florence, Sean Kirmani, Sumeet Singh, Vikas Sindhwani, Carolina Parada, Chelsea Finn, Peng Xu, Sergey Levine, Jie Tan

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；VLM(abstract)；分类 cs.AI

E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion

Ke Wang, Tianyu Xia, Zhangxuan Gu, Yi Zhao, Shuheng Shen, Changhua Meng, Weiqiang Wang, Ke Xu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments 9 pages, 5 figures, Under review

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu

专题命中 GUI与屏幕智能体 ：VLM(abstract)；grounding(abstract)；分类 cs.AI

Comments 51 pages, 21 figures

Human-Centered Automation

Carlos Toxtli

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments 12 pages, 0 figures

GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition

Guangzhao Dai, Xiangbo Shu, Wenhao Wu, Rui Yan, Jiachao Zhang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception

Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ICLR 2024 Workshop in Large Language Model (LLM) Agents

Language-Based Augmentation to Address Shortcut Learning in Object Goal Navigation

Dennis Hoftijzer, Gertjan Burghouts, Luuk Spreeuwers

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments 8 pages, 6 figures, to be published in IEEE IRC 2023

MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices

Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, Chunhua Shen

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Tech Report

ADriver-I: A General World Model for Autonomous Driving

Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, Tiancai Wang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments Tech Report

Towards Faithful Simulation of Human Shopping Behavior

面向人类购物行为的忠实模拟

Jiakai Tang, Yan Mi, Jing Yu, Yang Zhang, See-Kiong Ng, Qi Cao, Fei Sun, Xu Chen, Wen Chen, Jian Wu, Han Zhu, Bo Zheng

专题命中 GUI与屏幕智能体 ：VLM(abstract,abstract_cn)

AI总结 本文针对现有用户购物行为模拟器的记忆与优化挑战，提出分层记忆的RecVerse智能体，结合轨迹级RL优化，发布USB数据集，在模拟性能上显著优于现有基线。

UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents

UI-MOPD：用于持续GUI智能体学习的多平台策略蒸馏

Niu Lian, Tongbo Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Heng Qu, Shu-Tao Xia, Jinpeng Wang

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院） ； Xiaomi（小米） ； Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）） ； Zhejiang University（浙江大学） ； Peng Cheng Laboratory（鹏城实验室） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

AI总结 针对构建多平台GUI智能体的挑战，构建高质量数据集Uni-GUI，提出UI-MOPD方法，通过多教师策略蒸馏实现持续学习，动态选教师并转移行为先验，实验证明其能平衡跨平台能力保留与新平台适应。

Comments Technical report. 27 pages, 7 figures, 7 tables

MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization

MobileForge：基于分层反馈引导策略优化的移动GUI智能体免标注适配

Guangyi Liu, Pengxiang Zhao, Gao Wu, Yiwen Yin, Mading Li, Liang Liu, Congxiao Liu, Zhang Qi, Mengyan Wang, Liang Guo, Jiangning Zhang, Yong Liu

专题命中 GUI与屏幕智能体 ：MLLM(abstract,abstract_cn)

AI总结 提出MobileForge系统，通过MobileGym环境实现任务生成与评估，结合分层反馈引导策略优化（HiFPO）将轨迹结果、步骤反馈和修正提示转化为步骤级GRPO更新，实现移动GUI智能体免标注适配，在AndroidWorld上达到67.2% Pass@3。

Comments Project page: https://mobile-forge.github.io/

How Should Vision-Language-Action Models Use Proprioceptive State?

视觉-语言-动作（VLA）模型应如何使用本体感受状态？

Yiren Zhao, Ziyang Chen, Ziyang Rao, Pengteng Li, He Zhang, Weiyu Guo, Yandong Guo, Rushi Dai

专题命中 GUI与屏幕智能体 ：VLM(abstract,abstract_cn)

AI总结 本研究针对视觉-语言-动作（VLA）模型的本体感受状态接入方式开展受控实验，探究状态接入位置、历史长度等问题，得出系统性答案并提炼为可验证的VLA模型设计原则。

FreqNav: Stage-Wise Frequency Routing for Object-Oriented Aerial Vision-Language Navigation

FreqNav：面向面向对象空中视觉语言导航的分阶段频率路由

Yin Tang, Jiawei Ma, Jiahao Li, Hao Zhang, Zhemin Sun, Jianqiao Sun, Deyu Zhang

专题命中 GUI与屏幕智能体 ：grounding(abstract,abstract_cn)

AI总结 针对现有空中VLN方法的感知干扰问题，提出FreqNav框架，通过动态分配视觉令牌实现分阶段频率路由，提升性能并加快推理速度，验证了其实际应用潜力。

CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding

CycleVLA: 通过子任务回溯和最小贝叶斯风险解码实现的前瞻性自修正视觉-语言-动作模型

Chenyang Ma, Kai Lu, Guangyu Yang, Jiuming Liu, Shitong Xu, Bill Byrne, Ioannis Havoutis, Niki Trigoni, Andrew Markham

机构 * Department of Computer Science, University of Oxford（牛津大学计算机科学系） ； Department of Engineering, University of Cambridge（剑桥大学工程系） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract,abstract_cn)

AI总结 CycleVLA通过子任务回溯和最小贝叶斯风险解码实现前瞻性自修正，提升视觉-语言-动作模型的故障预测与恢复能力

Comments Project Page: https://dannymcy.github.io/cyclevla/

SOPD-SocialNav: Selective On-Policy Distillation for Vision-Language Social Navigation

SOPD-SocialNav：用于视觉语言社交导航的选择性在线策略蒸馏

Xinyu Zhang, Zishuo Wang, Ling Xiao

机构 * Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术研究生院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

AI总结 针对大规模视觉语言模型难部署、轻量级模型社交推理能力不足的问题，提出SOPD-SocialNav方法，通过基于熵的令牌选择机制及温度控制的散度目标，将社交导航知识从大型教师模型转移到轻量级学生模型，实验证明该方法有效。

Comments This paper has been accepted to 2026 WRC Symposium on Advanced Robotics and Automation (WRC SARA)

EdgeCoInfer: Hierarchical Collaborative Inference for On-Device Multimodal Large Models

EdgeCoInfer：用于设备端多模态大模型的分层协作推理

Lin Tan, Songtao Guo, Mingyan Li, David K. Y. Yau

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

AI总结 针对边缘环境中多模态大模型面临的挑战，提出EdgeCoInfer框架，通过模型间功能模块共享和模型内细粒度划分实现粒度自适应部署，采用混合进化分层强化学习解决问题，实验表明该框架能提高任务完成率并降低系统成本和内存消耗。

Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition

诊断智能体编排的视觉-语言-动作技能组合中的语义切换失败

Ke Rui, Yushen Zuo, Jiawei Wang, Haoran Jia, Jinming Ma, Weitao Zhou, Minglei Li

机构 * SimpleAI 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

AI总结 研究智能体编排的视觉-语言-动作技能组合中的语义切换失败问题，通过智能体编排执行框架，调用基于π0.5的技能检查点，在两种初始状态分布下评估，发现单技能与长期任务成功率差距大，为未来技能库改进提供诊断依据。

Journal ref RSS SemRob Workshop 2026

Let It Be Simple: One-Step Action Generation for Vision-Language-Action Models

让它简单：视觉-语言-动作模型的单步动作生成

Yitong Chen, Shiduo Zhang, Jingjing Gong, Xipeng Qiu

机构 * University of Science and Technology of China（中国科学技术大学） ； Shanghai Innovation Institute（上海创新研究院） ； Fudan University（复旦大学） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract_cn)；分类 cs.CV、cs.AI、cs.LG

AI总结 针对视觉-语言-动作（VLA）模型，提出通过偏置训练时间分布至高频噪声状态，实现无需教师模型、蒸馏或辅助目标的单步动作生成，性能可匹配十步解码。

Comments 13 pages, 10 figures

Open-Vocabulary Object-Goal Navigation by Generalizing Semantic Mapping with Dense CLIP

通过用密集CLIP泛化语义映射实现开放词汇目标导航

Meng Wei, Chenyang Wan, Tai Wang, Yuqiang Yang, Wenzhe Cai, Yilun Chen, Hanqing Wang, Jiangmiao Pang, Xihui Liu

机构 * Shanghai AI Lab（上海人工智能实验室） ； The University of Hong Kong（香港大学） ； Zhejiang University（浙江大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

AI总结 研究面向对象的实体导航任务，提出OVExp框架，利用密集CLIP模型展示基于语义地图的目标预测网络泛化能力，设计跨模态转移策略解决训练成本高问题，在ObjectNav基准上对未知目标有强大泛化能力。

Comments Accepted by ICRA 2026

CAC-VLA: Context-Gated Action Conditioning for Vision-Language-Action Models

CAC-VLA：用于视觉-语言-动作模型的上下文门控动作条件调节

Yifu Xiong, Wenhao Yu, Jiaxuan Lin, Bojun Zou, Jiahao Li, Lu Zhang, Yanyong Zhang, Jianmin Ji

机构 * University of Science and Technology of China (USTC)（中国科学技术大学） ； Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract,abstract_cn)

AI总结 研究视觉-语言-动作模型，提出上下文门控动作条件调节框架CAC-VLA，在视觉语言模型中学习轻量级潜在动作接口，训练模型预测潜在动作并通过上下文门调节动作专家，实验验证其有效性。

Comments 16 pages, 6 figures