Due to their ability of follow natural language instructions, vision-language-action (VLA) models are increasingly prevalent in the embodied AI arena, following the widespread success of their precursors -- LLMs and VLMs. In this paper, we discuss 10 principal milestones in the ongoing development of VLA models -- multimodality, reasoning, data, evaluation, cross-robot action generalization, efficiency, whole-body coordination, safety, agents, and coordination with humans. Furthermore, we discuss the emerging trends of using spatial understanding, modeling world dynamics, post training, and data synthesis -- all aiming to reach these milestones. Through these discussions, we hope to bring attention to the research avenues that may accelerate the development of VLA models into wider acceptability.
Flow Control: Steering Vision-Language-Action Models with Simple Real-Time Inputs
Departments of Electrical and Computer Engineering, Computer Science, and Neurobiology, University of California, Los Angeles(电气与计算机工程系、计算机科学系和神经生物学系,加州大学洛杉矶分校);Department of Computer Science University of California, Los Angeles(计算机科学系,加州大学洛杉矶分校)
Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach
Institute of Artificial Intelligence, China Telecom(中国电信人工智能研究院);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科学与技术大学)
TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models
ByteDance Seed(字节跳动种子);The University of Adelaide(阿德莱德大学);Tsinghua University(清华大学);Zhejiang University(浙江大学);The University of Hong Kong(香港大学);CSIRO Data61
Closed-Loop Neural Activation Control in Vision-Language-Action Models
Florida International University(佛罗里达国际大学);SRI International(美国桑尼沃德国际研究机构);United States Military Academy(美国军事学院);Oak Ridge National Laboratory(橡树岭国家实验室);University of Florida(佛罗里达大学)
Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
Carnegie Mellon University(卡内基梅隆大学);University of Southern California(南加州大学);University of California, Berkeley(加州大学伯克利分校);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
VLS: Steering Pretrained Robot Policies via Vision-Language Models
University of Washington(华盛顿大学);University of Oxford(牛津大学);National University of Singapore(新加坡国立大学);Allen Institute for Artificial Intelligence(人工智能研究院)
Vision-based Wearable Steering Assistance for People with Impaired Vision in Jogging
Institute of Advanced Technology, University of Science and Technology of China(中国科学技术大学先进技术研究院);School of Information and Electronic, Beijing Institute of Technology(北京理工大学信息与电子学院);School of Mechanical Engineering, Tongji University(同济大学机械工程学院)
本文提出部署于Jetson Orin NX的视觉可穿戴转向辅助系统,通过轻量多任务网络检测跑道线与障碍物,并结合采样和样条曲线规划,帮助视障者安全慢跑。
What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
University of Technology Nuremberg(纽伦堡工业大学);ETH Zürich(苏黎世联邦理工学院);Inria, Ecole Normale Supérieure, CNRS, PSL Research University(法国国家信息与自动化研究所,巴黎高等师范学校,法国国家科学研究中心,巴黎文理研究大学)
School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院);Shanghai Qizhi Institute(上海启智研究院);Shanghai Jiao Tong University(上海交通大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院)
TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
Shanghai Jiao Tong University(上海交通大学);Xense Robotics(Xense机器人公司);Sun Yat-sen University(中山大学);Oxford(牛津大学);Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);UCSD(加州大学圣地亚哥分校)