arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2507.04789cs.RO
收藏

无需训练的视觉语言模型(VLM)时间一致奖励生成

Training-free Generation of Temporally Consistent Rewards from VLMs

Beijing Institute of Technology(北京理工大学);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);Taobao & Tmall Group of Alibaba(阿里巴巴淘宝天猫集团);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)

Yinuo Zhao, Jiale Yuan, Zhiyuan Xu, Xiaoshuai Hao, Xinyi Zhang, Kun Wu, Zhengping Che, Chi Harold Liu, Jian Tang

AI总结:

针对VLMs在机器人操作中难以提供准确实时奖励的问题,提出无需训练的T²-VLM框架,通过跟踪VLM衍生子目标的状态变化生成时间一致奖励,提升了长时决策能力和故障恢复能力,在基准测试中表现优异。

摘要:

视觉语言模型(VLMs)的最新进展显著提升了目标分解、视觉理解等具身任务的性能。但因预训练数据集缺乏特定领域机器人知识,且计算成本高阻碍实时应用,在不微调VLMs的情况下为机器人操作提供准确奖励仍具挑战。为此,我们提出T²-VLM——一种无需训练、时间一致的新型框架,通过跟踪VLM衍生子目标的状态变化生成准确奖励。具体而言,每次交互前,我们先查询VLM以建立空间感知子目标和初始完成度估计;然后采用贝叶斯跟踪算法,利用子目标隐藏状态动态更新完成状态,生成结构化奖励。该框架增强了长时决策能力,提升了强化学习(RL)智能体的故障恢复能力。实验表明,T²-VLM在两个机器人操作基准中达到了最先进性能,奖励准确率更高且计算消耗更低。

英文摘要:

Recent advances in vision-language models (VLMs) have significantly improved performance in embodied tasks such as goal decomposition and visual comprehension. However, providing accurate rewards for robotic manipulation without fine-tuning VLMs remains challenging due to the absence of domain-specific robotic knowledge in pre-trained datasets and high computational costs that hinder real-time applicability. To address this, we propose $\mathrm{T}^2$-VLM, a novel training-free, temporally consistent framework that generates accurate rewards through tracking the status changes in VLM-derived subgoals. Specifically, our method first queries the VLM to establish spatially aware subgoals and an initial completion estimate before each round of interaction. We then employ a Bayesian tracking algorithm to update the goal completion status dynamically, using subgoal hidden states to generate structured rewards for reinforcement learning (RL) agents. This approach enhances long-horizon decision-making and improves failure recovery capabilities with RL. Extensive experiments indicate that $\mathrm{T}^2$-VLM achieves state-of-the-art performance in two robot manipulation benchmarks, demonstrating superior reward accuracy with reduced computation consumption. We believe our approach not only advances reward generation techniques but also contributes to the broader field of embodied AI. Project website: https://t2-vlm.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑