The strong performance of large vision-language models (VLMs) trained with reinforcement learning (RL) has motivated similar approaches for fine-tuning vision-language-action (VLA) models in robotics. Many recent works fine-tune VLAs directly in the real world to avoid addressing the sim-to-real gap. While real-world RL circumvents sim-to-real issues, it inherently limits the generality of the resulting VLA, as scaling scene and object diversity in the physical world is prohibitively difficult. This leads to the paradoxical outcome of transforming a broadly pretrained model into an overfitted, scene-specific policy. Training in simulation can instead provide access to diverse scenes, but designing those scenes is also costly. In this work, we show that VLAs can be RL fine-tuned across broad scene and object distributions and with reduced labor by leveraging 3D world generative models. Using these models together with a language-driven scene designer, we generate 100 diverse interactive scenes containing unique objects and backgrounds, enabling scalable and highly parallel policy learning. Starting from a pretrained imitation baseline, our approach increases simulation success from 9.7% up to 79.8% while achieving a 1.25$\times$ speedup in task completion time. We further demonstrate successful sim-to-real transfer enabled by the quality of the generated scenes together with domain randomization, improving real-world success from 21.7% to 75% and achieving a 1.13$\times$ speedup. Finally, we further highlight the benefits of leveraging the effectively unlimited data from 3D world generative models through an ablation study showing that increasing scene diversity directly improves zero-shot generalization.
Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
University of Applied Science and Arts of Southern Switzerland, Department of Innovative Technologies(瑞士南方应用科学与艺术大学创新技术系);Università della Svizzera Italiana, Faculty of Informatics, Lugano, Switzerland(瑞士意大利大学信息学院,卢加诺,瑞士)
SA-VLA: Spatially-Aware Flow-Matching for Vision-Language-Action Reinforcement Learning
Wuhan University(武汉大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);National University of Singapore(新加坡国立大学);Chinese Academy of Sciences(中国科学院);Nanyang Technological University(南洋理工大学)
VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电子与电气工程学院)
Sample-Efficient Robot Skill Learning for Construction Tasks: Benchmarking Hierarchical Reinforcement Learning and Vision-Language-Action VLA Model
Department of Civil Engineering, Stony Brook University(土木工程系,石溪大学);Department of Civil and Environmental Engineering, Virginia Tech(土木与环境工程系,弗吉尼亚理工大学);Department of Electrical and Computer Engineering, Virginia Tech(电气与计算机工程系,弗吉尼亚理工大学)
3D-VLA: A 3D Vision-Language-Action Generative World Model
University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Shanghai Jiao Tong University(上海交通大学);South China University of Technology(华南理工大学);Wuhan University(武汉大学);Massachusetts Institute of Technology(麻省理工学院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation
Institute of Automation, Chinese Academy of Sciences, Beijing, China(中国科学院自动化研究所);Beijing Academy of Artificial Intelligence, Beijing, China(北京人工智能研究院);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China(北京大学计算机科学学院多媒体信息处理国家重点实验室)
RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Models
State Key Laboratory of Mechanical System and Vibration(机械系统与振动国家重点实验室);Shanghai Key Laboratory of Intelligent Robotics(上海智能机器人重点实验室);School of Mechanical Engineering, Shanghai Jiao Tong University(上海交通大学机械工程学院)
Learning from Hindsight for VLA Reinforcement Learning
Massachusetts Institute of Technology(麻省理工学院);MIT-IBM Computing Research Lab(麻省理工学院-IBM计算研究实验室);Stanford University(斯坦福大学);University of California, San Diego(加利福尼亚大学圣地亚哥分校)
Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);SenseTime Research(商汤科技研究院);Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学)
Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
University of Toronto Robotics Institute(多伦多大学机器人研究所);Vector Institute(向量研究所);Montreal Institute of Learning Algorithms (MILA)(蒙特利尔学习算法研究所);McGill University(麦吉尔大学)
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
Sun Yat-sen University(中山大学);Shenzhen Loop Area Institute(深圳河套学院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai AI Laboratory(上海人工智能实验室);University of Oxford(牛津大学);University of Science and Technology of China(中国科学技术大学)
Pure Vision Language Action (VLA) Models: A Comprehensive Survey
Lanzhou University, China(兰州大学);National University of Singapore, Singapore(新加坡国立大学);Institute of Computing Technology, Chinese Academy of Sciences, China(中国科学院计算技术研究所);NExT++ Research Centre, National University of Singapore, Singapore(新加坡国立大学NExT++研究中心)