视觉大模型 / VLM

PADriver: Towards Personalized Autonomous Driving

Genghua Kou, Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Ziheng Zhang, Osamu Yoshie, Tiancai Wang, Ying Li, Xiangyu Zhang

机构 * Beijing Institute of Technology（北京理工大学） ； Megvii Technology（商汤科技） ； Waseda University（早稻田大学） 纠错

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV

Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation

Sonia Raychaudhuri, Duy Ta, Katrina Ashton, Angel X. Chang, Jiuguang Wang, Bernadette Bucher

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

ScaleTrack: Scaling and back-tracking Automated GUI Agents

Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, Lin Ma

机构 * Meituan（美团） ； Zhejiang University（浙江大学） ； University of Adelaide（阿德莱德大学） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

UFO2: The Desktop AgentOS

Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, Liqun Li, Yu Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

机构 * Microsoft（微软公司） ； ZJU-UIUC Institute（浙大-伊利诺伊大学联合研究所） ； Nanjing University（南京大学） ； Peking University（北京大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

Comments The source code of UFO2 is publicly available at https://github.com/microsoft/UFO/, with comprehensive documentation provided at https://microsoft.github.io/UFO/

Cracking the Code of Action: a Generative Approach to Affordances for Reinforcement Learning

Lynn Cherif, Flemming Kondrup, David Venuto, Ankit Anand, Doina Precup, Khimya Khetarpal

机构 * McGill University（麦吉尔大学） ； Mila ； Google DeepMind（谷歌DeepMind） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation

Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, Shanghang Zhang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

Is Temporal Prompting All We Need For Limited Labeled Action Recognition?

Shreyank N Gowda, Boyan Gao, Xiao Gu, Xiaobo Jin

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted in CVPR-W 2025

Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

Xuesong Zhang, Jia Li, Yunbo Xu, Zhenzhen Hu, Richang Hong

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments 8 pages, 5 figures, accepted at IJCNN 2025

Multi-modal Knowledge Distillation-based Human Trajectory Forecasting

Jaewoo Jeong, Seohee Lee, Daehee Park, Giwon Lee, Kuk-Jin Yoon

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV

Comments Accepted to CVPR 2025

Progress-Aware Video Frame Captioning

Zihui Xue, Joungbin An, Xitong Yang, Kristen Grauman

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV

Comments Accepted by CVPR 2025, Project website: https://vision.cs.utexas.edu/projects/ProgressCaptioner/

ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, Xiang Bai

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulation

Chenxuan Li, Jiaming Liu, Guanqun Wang, Xiaoqi Li, Sixiang Chen, Liang Heng, Chuyan Xiong, Jiaxin Ge, Renrui Zhang, Kaichen Zhou, Shanghang Zhang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues

Francesco Taioli, Edoardo Zorzi, Gianni Franchi, Alberto Castellini, Alessandro Farinelli, Marco Cristani, Yiming Wang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

Comments https://intelligolabs.github.io/CoIN/

Zero-Shot Action Recognition in Surveillance Videos

Joao Pereira, Vasco Lopes, David Semedo, Joao Neves

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling

Tsu-Jui Fu, Xin Eric Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, William Yang Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments ECCV'20 (Spotlight)

MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Models

Han Zhao, Wenxuan Song, Donglin Wang, Xinyang Tong, Pengxiang Ding, Xuelian Cheng, Zongyuan Ge

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.AI

Comments Accepted by ICRA 2025

VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation

Hritik Bansal, Clark Peng, Yonatan Bitton, Roman Goldenberg, Aditya Grover, Kai-Wei Chang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments 41 pages, 33 Figures

VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model

Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.LG

Comments 20pages,5 figures

Digi-Q: Learning Q-Value Functions for Training Device-Control Agents

Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, Aviral Kumar

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.LG

Comments Accepted to ICLR 2025

PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC

Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, Fei Huang

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV

Comments 14 pages, 7 figures

NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants

Yiran Qin, Ao Sun, Yuze Hong, Benyou Wang, Ruimao Zhang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted to ICRA2025

SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World

Jiaqi Zhang, Chen Gao, Liyuan Zhang, Yong Li, Hongzhi Yin

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

GUI Agents with Foundation Models: A Comprehensive Survey

Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang, Chuhan Wu, Yasheng Wang, Ruiming Tang, Jianye Hao

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding

Jiaxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Weixuan chen, Xihan Wei, Liefeng Bo

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Large Action Models: From Inception to Implementation

Lu Wang, Fangkai Yang, Chaoyun Zhang, Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang, Si Qin, Qisheng Su, Jiayi Ye, Yudi Zhang, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments 25pages,12 figures

Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation

Senwei Xie, Hongyu Wang, Zhanqi Xiao, Ruiping Wang, Xilin Chen

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Large Language Models for Video Surveillance Applications

Ulindu De Silva, Leon Fernando, Billy Lau Pik Lik, Zann Koh, Sam Conrad Joyce, Belinda Yuen, Chau Yuen

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV

Comments Accepted for TENCON 2024

GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent

Kangjia Zhao, Jiahui Song, Leigang Sha, Haozhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, Jianwei Yin

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs

Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, Michael R. Lyu

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.AI

MobileFlow: A Multimodal LLM For Mobile GUI Agent

Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, Wenhao Xu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV