视觉大模型 / VLM

GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking

Kristian Kolthoff, Felix Kretzer, Christian Bartelt, Alexander Maedche, Simone Paolo Ponzetto

专题命中 GUI与屏幕智能体 ：MLLM(abstract)

CookBench: A Long-Horizon Embodied Planning Benchmark for Complex Cooking Scenarios

Muzhen Cai, Xiubo Chen, Yining An, Jiaxin Zhang, Xuesong Wang, Wang Xu, Weinan Zhang, Ting Liu

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments 9 pages, 5 figures

SA-GCS: Semantic-Aware Gaussian Curriculum Scheduling for UAV Vision-Language Navigation

Hengxing Cai, Jinhan Dong, Yijie Rao, Jingcheng Deng, Jingjun Tan, Qien Chen, Haidong Wang, Zhen Wang, Shiyu Huang, Agachai Sumalee, Renxin Zhong

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

FMimic: Foundation Models are Fine-grained Action Learners from Human Videos

Guangyan Chen, Meiling Wang, Te Cui, Yao Mu, Haoyang Lu, Zicai Peng, Mengxiao Hu, Tianxing Zhou, Mengyin Fu, Yi Yang, Yufeng Yue

机构 * Beijing Institute of Technology, Beijing, P. R. China（北京理工大学） ； The University of Hong Kong, Hong Kong, P. R. China（香港大学） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)

Comments accepted to International Journal of Robotics Research(IJRR)

MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation

Ning Li, Xiangmou Qu, Jiamu Zhou, Jun Wang, Muning Wen, Kounianhua Du, Xingyu Lou, Qiuying Peng, Jun Wang, Weinan Zhang

机构 * Shanghai Jiao Tong University（上海交通大学） ； OPPO Research Institute（OPPO研究院） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments A technical report on a GUI agent based on multi-agent systems

VL-Explore: Zero-shot Vision-Language Exploration and Target Discovery by Mobile Robots

Yuxuan Zhang, Adnan Abdullah, Sanjeev J. Koppal, Md Jahidul Islam

机构 * RoboPI Laboratory, Dept. of ECE, University of Florida (UF)（罗波实验室，电子工程系，佛罗里达大学） ； FOCUS Laboratory, Dept. of ECE, University of Florida (UF)（焦点实验室，电子工程系，佛罗里达大学） ； Amazon Robotics（亚马逊机器人技术） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments V2, includes suppl as appendix

EdgeVLA: Efficient Vision-Language-Action Models

Paweł Budzianowski, Wesley Maa, Matthew Freed, Jingxiang Mo, Winston Hsiao, Aaron Xie, Tomasz Młoduchowski, Viraj Tipnis, Benjamin Bolte

机构 * K-Scale Labs（K-Scale实验室） ； McGill University（麦吉尔大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Journal ref IROS-MoMA3 Workshop 2024

MOSU: Autonomous Long-range Robot Navigation with Multi-modal Scene Understanding

Jing Liang, Kasun Weerakoon, Daeun Song, Senthurbavan Kirubaharan, Xuesu Xiao, Dinesh Manocha

机构 * University of Maryland, College Park MD, 20740, USA（马里兰大学） ； Goerge Mason University, Fairfax, VA, 22030, USA（乔治·马歇尔大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments

Amirreza Payandeh, Anuj Pokhrel, Daeun Song, Marcos Zampieri, Xuesu Xiao

机构 * Department of Computer Science, George Mason University（计算机科学系，乔治·马歇尔大学） ； Department of Information Sciences and Technology, George Mason University（信息科学与技术系，乔治·马歇尔大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

DesignCoder: Hierarchy-Aware and Self-Correcting UI Code Generation with Large Language Models

Yunnong Chen, Shixian Ding, YingYing Zhang, Wenkai Chen, Jinzhou Du, Lingyun Sun, Liuqing Chen

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments 11 pages,6 figures

RationalVLA: A Rational Vision-Language-Action Model with Dual System

Wenxuan Song, Jiayi Chen, Wenxue Li, Xu He, Han Zhao, Can Cui, Pengxiang Ding Shiyan Su, Feilong Tang, Xuelian Cheng, Donglin Wang, Zongyuan Ge, Xinhu Zheng, Zhe Liu, Hesheng Wang, Haoang Li

机构 * The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； Westlake University（西湖大学） ； Monash University（墨尔本大学） ； Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments 14 pages

GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs

Ruofan Liu, Xiwen Teoh, Yun Lin, Guanjie Chen, Ruofei Ren, Denys Poshyvanyk, Jin Song Dong

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization

Jonathan Yang, Chuyuan Kelly Fu, Dhruv Shah, Dorsa Sadigh, Fei Xia, Tingnan Zhang

机构 * Stanford University（斯坦福大学） ； Google DeepMind（谷歌DeepMind） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)

Comments 16 pages, 13 figures

FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents

Bobo Li, Yuheng Wang, Hao Fei, Juncheng Li, Wei Ji, Mong-Li Lee, Wynne Hsu

机构 * National University of Singapore（新加坡国立大学） ； Wuhan University（武汉大学） ； Zhejiang University（浙江大学） ； Nanjing University（南京大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments 8 pages, 7 figures

UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs

Yi Gui, Zhen Li, Zhongyi Zhang, Yao Wan, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen, Xing Zhou, Wenbin Jiang, Xiangliang Zhang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments WWW' 2025

CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding

Wenxuan Ma, Xiaoge Cao, Yixiang Zhang, Chaofan Zhang, Shaobo Yang, Peng Hao, Bin Fang, Yinghao Cai, Shaowei Cui, Shuo Wang

机构 * Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； Beihang University（北航） ； Beijing University of Posts and Telecommunications（北京邮电大学） ； Samsung Research China（三星中国研究院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments 16 pages

LogisticsVLN: Vision-Language Navigation For Low-Altitude Terminal Delivery Based on Agentic UAVs

Xinyuan Zhang, Yonglin Tian, Fei Lin, Yue Liu, Jing Ma, Kornélia Sára Szatmáry, Fei-Yue Wang

机构 * School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） ； State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室） ； Department of Engineering Science, Faculty of Innovation Engineering, Macau University of Science and Technology（澳门科技大学创新工程学院工程科学系） ； China Ship Research and Development Academy（中国船舶科研 Academy） ； Obuda University（奥布达大学） ； State Key Laboratory for Management and Control of Complex Systems, Chinese Academy of Sciences（中国科学院复杂系统管理与控制国家重点实验室） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, Caiming Xiong

专题命中 GUI与屏幕智能体 ：grounding(abstract)

Comments ICML 2025

Media Content Atlas: A Pipeline to Explore and Investigate Multidimensional Media Space using Multimodal LLMs

Merve Cerit, Eric Zelikman, Mu-Jung Cho, Thomas N. Robinson, Byron Reeves, Nilam Ram, Nick Haber

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments Accepted to CHI 2025, in press. See the project page at mediacontentatlas.github.io

Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development

Sabrina Haque, Christoph Csallner

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction

Yiqiao Jin, Stefano Petrangeli, Yu Shen, Gang Wu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments Accepted to MM4SG Workshop at The Web Conference 2025

VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulation

Wei Zhao, Pengxiang Ding, Min Zhang, Zhefei Gong, Shuanghao Bai, Han Zhao, Donglin Wang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments Accepted as a conference paper at ICLR 2025

OpenBench: A New Benchmark and Baseline for Semantic Navigation in Smart Logistics

Junhui Wang, Dongjie Huo, Zehui Xu, Yongliang Shi, Yimin Yan, Yuanxin Wang, Chao Gao, Yan Qiao, Guyue Zhou

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Mobile Manipulation Instruction Generation from Multiple Images with Automatic Metric Enhancement

Kei Katsumata, Motonari Kambara, Daichi Yashima, Ryosuke Korekata, Komei Sugiura

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments Accepted for IEEE RA-L 2025

Sketch-MoMa: Teleoperation for Mobile Manipulator via Interpretation of Hand-Drawn Sketches

Kosei Tanada, Yuka Iwanaga, Masayoshi Tsuchinaga, Yuji Nakamura, Takemitsu Mori, Remi Sakai, Takashi Yamamoto

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments This work has been submitted to the IEEE for possible publication. Project Page: https://toyotafrc.github.io/SketchMoMa-Proj

Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models

Linus Nwankwo, Elmar Rueckert

专题命中 GUI与屏幕智能体 ：visual language model(abstract)

A model-based approach for transforming InSAR-derived vertical land motion from a local to a global reference frame

Mahmoud Reshadati, Manoochehr Shirzaei

专题命中 GUI与屏幕智能体 ：VLM(abstract)

Comments 13 pages, 7 figures

RoLD: Robot Latent Diffusion for Multi-task Policy Modeling

Wenhui Tan, Bei Liu, Junbo Zhang, Ruihua Song, Jianlong Fu

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Context-Aware Replanning with Pre-explored Semantic Map for Object Navigation

Po-Chen Ko, Hung-Ting Su, Ching-Yuan Chen, Jia-Fong Yeh, Min Sun, Winston H. Hsu

专题命中 GUI与屏幕智能体 ：visual language model(abstract)

Comments CoRL 2024 camera ready. The first three authors contributed equally, and their order of authorship is interchangeable. Project page: https://care-maps.github.io/

A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter

Kechun Xu, Shuqi Zhao, Zhongxiang Zhou, Zizhang Li, Huaijin Pi, Yue Wang, Rong Xiong

专题命中 GUI与屏幕智能体 ：grounding(abstract)

Comments Accepted by ICRA 2023