arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03973cs.ROcs.CV
收藏

VLS:通过视觉-语言模型引导预训练的机器人策略

VLS: Steering Pretrained Robot Policies via Vision-Language Models

University of Washington(华盛顿大学);University of Oxford(牛津大学);National University of Singapore(新加坡国立大学);Allen Institute for Artificial Intelligence(人工智能研究院)

Shuo Liu, Ishneet Sukhvinder Singh, Yiqing Xu, Jiafei Duan, Ranjay Krishna

AI总结:

VLS通过视觉-语言模型在推理时引导预训练机器人策略,实现无需训练的适应,提升在不同环境下的表现。

摘要:

为什么预训练的扩散或流匹配策略在执行相同任务时接近障碍物、在移位支撑面上或在轻微杂乱环境中会失败?此类失败很少反映缺失的运动技能;相反,它们暴露了在训练-测试转移中模仿学习的局限性,其中动作生成紧密耦合到训练特定的空间配置和任务规范。为了解决这些失败而重新训练或微调是昂贵且概念上不正确的,因为所需的行为已经存在,但无法在测试时选择性适应。我们提出了视觉-语言引导(VLS),一种无需训练的框架,用于冻结生成机器人策略的推理时间适应。VLS将适应视为推理时间的控制问题,引导预训练扩散或流匹配策略的采样过程,响应于分布外的观测-语言输入,而无需修改策略参数。通过利用视觉-语言模型合成轨迹可微奖励函数,VLS引导去噪过程朝着满足测试时间空间和任务要求的动作轨迹。在模拟和现实世界评估中,VLS在模拟和现实世界中均优于先前的引导方法,实现了在CALVIN上的31%提升和在LIBERO-PRO上的13%提升。在Franka机器人上的现实世界部署进一步展示了在测试时间空间和语义转移下的鲁棒推理时间适应。项目页面:https://vision-language-steering.github.io/webpage/

英文摘要:

Why do pretrained diffusion or flow-matching policies fail when the same task is performed near an obstacle, on a shifted support surface, or amid mild clutter? Such failures rarely reflect missing motor skills; instead, they expose a limitation of imitation learning under train-test shifts, where action generation is tightly coupled to training-specific spatial configurations and task specifications. Retraining or fine-tuning to address these failures is costly and conceptually misaligned, as the required behaviors already exist but cannot be selectively adapted at test time. We propose Vision-Language Steering (VLS), a training-free framework for inference-time adaptation of frozen generative robot policies. VLS treats adaptation as an inference-time control problem, steering the sampling process of a pretrained diffusion or flow-matching policy in response to out-of-distribution observation-language inputs without modifying policy parameters. By leveraging vision-language models to synthesize trajectory-differentiable reward functions, VLS guides denoising toward action trajectories that satisfy test-time spatial and task requirements. Across simulation and real-world evaluations, VLS consistently outperforms prior steering methods, achieving a 31% improvement on CALVIN and a 13% gain on LIBERO-PRO. Real-world deployment on a Franka robot further demonstrates robust inference-time adaptation under test-time spatial and semantic shifts. Project page: https://vision-language-steering.github.io/webpage/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑