VLS: Steering Pretrained Robot Policies via Vision-Language Models
University of Washington(华盛顿大学);University of Oxford(牛津大学);National University of Singapore(新加坡国立大学);Allen Institute for Artificial Intelligence(人工智能研究院)
Why do pretrained diffusion or flow-matching policies fail when the same task is performed near an obstacle, on a shifted support surface, or amid mild clutter? Such failures rarely reflect missing motor skills; instead, they expose a limitation of imitation learning under train-test shifts, where action generation is tightly coupled to training-specific spatial configurations and task specifications. Retraining or fine-tuning to address these failures is costly and conceptually misaligned, as the required behaviors already exist but cannot be selectively adapted at test time. We propose Vision-Language Steering (VLS), a training-free framework for inference-time adaptation of frozen generative robot policies. VLS treats adaptation as an inference-time control problem, steering the sampling process of a pretrained diffusion or flow-matching policy in response to out-of-distribution observation-language inputs without modifying policy parameters. By leveraging vision-language models to synthesize trajectory-differentiable reward functions, VLS guides denoising toward action trajectories that satisfy test-time spatial and task requirements. Across simulation and real-world evaluations, VLS consistently outperforms prior steering methods, achieving a 31% improvement on CALVIN and a 13% gain on LIBERO-PRO. Real-world deployment on a Franka robot further demonstrates robust inference-time adaptation under test-time spatial and semantic shifts. Project page: https://vision-language-steering.github.io/webpage/
A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detection
Aerospace and Informatics Domain(航空航天与信息领域);National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing(空间智能信息处理国家级重点实验室);School of Electronic(电子学院)
Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learning
The University of Hong Kong(香港大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);Huawei Cloud Computing Technologies(华为云计算技术);Ola Dimensions;HKU Shanghai Intelligent Computing Research Center(香港大学上海智能计算研究中心)
School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院);Shanghai Qizhi Institute(上海启智研究院);Shanghai Jiao Tong University(上海交通大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院)
Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents
Tsinghua University(清华大学);Striding AI(驰行人工智能公司);Purdue University(普渡大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Infinigence AI(英飞智研人工智能公司);Zhongguancun Academy(中关村学院);Hong Kong University of Science and Technology(香港科技大学)