视觉大模型 / VLM

RAVEN: Resilient Aerial Navigation via Open-Set Semantic Memory and Behavior Adaptation

Seungchan Kim, Omar Alama, Dmytro Kurdydyk, John Keller, Nikhil Keetha, Wenshan Wang, Yonatan Bisk, Sebastian Scherer

机构 * Carnegie Mellon University（卡内基梅隆大学） ； Davidson College（戴维森学院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Do We Need Large VLMs for Spotting Soccer Actions?

Ritabrata Chakraborty, Rajatsubhra Chakraborty, Avijit Dasgupta, Sandeep Chaurasia

机构 * Manipal University Jaipur（贾浦尔曼普尔大学） ； UNC–Charlotte（北卡罗来纳州立大学夏洛特分校） ； IIIT Hyderabad（海得拉巴印度理工学院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 6 pages, 2 tables

ComposableNav: Instruction-Following Navigation in Dynamic Environments via Composable Diffusion

Zichao Hu, Chen Tang, Michael J. Munje, Yifeng Zhu, Alex Liu, Shuijing Liu, Garrett Warnell, Peter Stone, Joydeep Biswas

机构 * Department of Computer Science, The University of Texas at Austin（德克萨斯大学计算机科学系） ； Army Research Laboratory（陆军研究实验室） ； Sony AI（索尼人工智能） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Conference on Robot Learning (CoRL) 2025 Project site: https://amrl.cs.utexas.edu/ComposableNav/

Serendipitous Recommendation with Multimodal LLM

Haoting Wang, Jianling Wang, Hao Li, Fangjun Yi, Mengyu Fu, Youwei Zhang, Yifan Liu, Liang Liu, Minmin Chen, Ed H. Chi, Lichan Hong, Haokai Lu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Comments Accepted by 2025 Recsys EARL Workshop

CollabVLA: Self-Reflective Vision-Language-Action Model Dreaming Together with Human

Nan Sun, Yongchang Li, Chenxu Wang, Huiying Li, Huaping Liu

机构 * Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；grounding(abstract)

Comments 8 pages, 5 figures, 3 tables

Handle Object Navigation as Weighted Traveling Repairman Problem

Ruimeng Liu, Xinhang Xu, Shenghai Yuan, Lihua Xie

机构 * Centre for Advanced Robotics Technology Innovation (CARTIN), School of Electrical and Electronic Engineering, Nanyang Technological University（先进机器人技术创新中心（CARTIN）、电子与电气工程学院、南洋理工大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations

Shresth Grover, Akshay Gopalkrishnan, Bo Ai, Henrik I. Christensen, Hao Su, Xuanlin Li

机构 * UC San Diego（圣迭戈大学） ； Hillbot 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project Page: https://gen-vla.github.io/

Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions

Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, Hai Zhao

机构 * School of Computer Science（计算机学院） ； Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering（智能交互与认知工程重点实验室） ； Shanghai Jiao Tong University（上海交通大学） ； Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3（Web3可信数据流通与治理上海市重点实验室） ； GenAI, Meta（Meta GenAI） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Comments ACL 2025

MoTo: A Zero-shot Plug-in Interaction-aware Navigation for General Mobile Manipulation

Zhenyu Wu, Angyuan Ma, Xiuwei Xu, Hang Yin, Yinan Liang, Ziwei Wang, Jiwen Lu, Haibin Yan

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

Comments Accepted to CoRL 2025. Project Page: https://gary3410.github.io/MoTo/

GeoVLA: Empowering 3D Representations in Vision-Language-Action Models

Lin Sun, Bin Xie, Yingfei Liu, Hao Shi, Tiancai Wang, Jiale Cao

机构 * Tianjin University（天津大学） ； Dexmal ； Tsinghua University（清华大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

Comments The project is visible at https://linsun449.github.io/GeoVLA/

SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience

Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, Jiaqi Wang

机构 * Shanghai Jiao Tong University（上海交通大学） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Code at https://github.com/SunzeY/SEAgent

VisioPath: Vision-Language Enhanced Model Predictive Control for Safe Autonomous Navigation in Mixed Traffic

Shanting Wang, Panagiotis Typaldos, Chenjun Li, Andreas A. Malikopoulos

机构 * System Engineering Program, Cornell University（Cornell大学系统工程项目） ； School of Civil and Environmental Engineering, Cornell University（Cornell大学土木与环境工程学院） ； School of Electrical and Computer Engineering, Cornell University（Cornell大学电气与计算机工程学院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

DyNaVLM: Zero-Shot Vision-Language Navigation System with Dynamic Viewpoints and Self-Refining Graph Memory

Zihe Ji, Huangxuan Lin, Yue Gao

机构 * SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University（上海交通大学巴黎精英技术研究所） ； Department of Automation, Shanghai Jiao Tong University（上海交通大学自动化系） ； MoE Key Lab of Artificial Intelligence and AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能与AI研究所） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

Can Pretrained Vision-Language Embeddings Alone Guide Robot Navigation?

Nitesh Subedi, Adam Haroon, Shreyan Ganguly, Samuel T. K. Tetteh, Prajwal Koirala, Cody Fleming, Soumik Sarkar

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

Comments 6 figures, 2 tables, Accepted to Robotics: Science and Systems (RSS) 2025 Workshop on Robot Planning in the Era of Foundation Models (FM4RoboPlan)

Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

Yuan Guo, Tingjia Miao, Zheng Wu, Pengzhou Cheng, Ming Zhou, Zhuosheng Zhang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；multimodal large language model(abstract)

Multimodal Spatial Language Maps for Robot Navigation and Manipulation

Chenguang Huang, Oier Mees, Andy Zeng, Wolfram Burgard

机构 * University of Technology Nuremberg（图恩堡技术大学） ； UC Berkeley（伯克利大学） ； Google Research（谷歌研究） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments accepted to International Journal of Robotics Research (IJRR). 24 pages, 18 figures. The paper contains texts from VLMaps(arXiv:2210.05714) and AVLMaps(arXiv:2303.07522). The project page is https://mslmaps.github.io/

BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation

Zibo Zhou, Yue Hu, Lingkai Zhang, Zonglin Li, Siheng Chen

机构 * Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

MiMo-VL Technical Report

Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhixian Zheng, Zhichao Song, Zhenbo Luo, Yue Yu, Yudong Wang, Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huang, Xu Wang, Xinzhe Xu, Xingchen Song, Xing Zhang, Xing Yong, Xin Zhang, Xiangwei Deng, Wenyu Yang, Wenhan Ma, Weiwei Lv, Weiji Zhuang, Wei Liu, Sirui Deng, Shuo Liu, Shimao Chen, Shihua Yu, Shaohui Liu, Shande Wang, Rui Ma, Qiantong Wang, Peng Wang, Nuo Chen, Menghang Zhu, Kangyang Zhou, Kang Zhou, Kai Fang, Jun Shi, Jinhao Dong, Jiebao Xiao, Jiaming Xu, Huaqiu Liu, Hongshen Xu, Heng Qu, Haochen Zhao, Hanglong Lv, Guoan Wang, Duo Zhang, Dong Zhang, Di Zhang, Chong Ma, Chang Liu, Can Cai, Bingquan Xia

机构 * LLM-Core ； Xiaomi（小米） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

Comments 32 pages

LifeIR at the NTCIR-18 Lifelog-6 Task

Jiahan Chen, Da Li, Keping Bi

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

Yunseok Jang, Yeda Song, Sungryull Sohn, Lajanugen Logeswaran, Tiange Luo, Dong-Ki Kim, Kyunghoon Bae, Honglak Lee

机构 * University of Michigan（密歇根大学） ； LG AI Research（LG人工智能研究） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments CVPR 2025

BadNAVer: Exploring Jailbreak Attacks On Vision-and-Language Navigation

Wenqi Lyu, Zerui Li, Yanyuan Qiao, Qi Wu

机构 * Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Comments 8 pages, 4 figures

VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation

Chaofan Zhang, Peng Hao, Xiaoge Cao, Xiaoshuai Hao, Shaowei Cui, Shuo Wang

机构 * State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院） ； Samsung R&D Institute China–Beijing（三星中国北京研发中心） ； Beijing Academy of Artificial Intelligence（北京人工智能研究院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

UAV-VLPA*: A Vision-Language-Path-Action System for Optimal Route Generation on a Large Scales

Oleg Sautenkov, Aibek Akhmetkazy, Yasheerah Yaqoot, Muhammad Ahsan Mustafa, Grik Tadevosyan, Artem Lykov, Dzmitry Tsetserukou

机构 * Intelligent Space Robotics Laboratory, Center for Digital Engineering, Skolkovo Institute of Science and Technology（智能空间机器人实验室，数字工程中心，斯克尔科沃科学与技术研究所） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；visual language model(abstract)

Comments arXiv admin note: text overlap with arXiv:2501.05014

Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success

Moo Jin Kim, Chelsea Finn, Percy Liang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted to Robotics: Science and Systems (RSS) 2025. Project website: https://openvla-oft.github.io/

SceneScout: Towards AI Agent-driven Access to Street View Imagery for Blind Users

Gaurav Jain, Leah Findlater, Cole Gleason

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents

Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, Xin Eric Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 18 pages, 13 figures, 8 tables

DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents

Yibin Xu, Liang Yang, Hao Chen, Hua Wang, Zhi Chen, Yaohua Tang

专题命中 GUI与屏幕智能体 ：visual language model(abstract)；grounding(abstract)

AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu

专题命中 GUI与屏幕智能体 ：VLM(abstract)；grounding(abstract)

Comments ICLR2025 Spotlight https://agenttrek.github.io

Magma: A Foundation Model for Multimodal AI Agents

Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 29 pages, 16 figures, technical report from MSR

Leveraging Multimodal LLM for Inspirational User Interface Search

Seokhyeon Park, Yumin Song, Soohyun Lee, Jaeyoung Kim, Jinwook Seo

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Comments In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '25)