Training-free Generation of Temporally Consistent Rewards from VLMs
Beijing Institute of Technology(北京理工大学);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);Taobao & Tmall Group of Alibaba(阿里巴巴淘宝天猫集团);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
Yinuo Zhao, Jiale Yuan, Zhiyuan Xu, Xiaoshuai Hao, Xinyi Zhang, Kun Wu, Zhengping Che, Chi Harold Liu, Jian Tang
Recent advances in vision-language models (VLMs) have significantly improved performance in embodied tasks such as goal decomposition and visual comprehension. However, providing accurate rewards for robotic manipulation without fine-tuning VLMs remains challenging due to the absence of domain-specific robotic knowledge in pre-trained datasets and high computational costs that hinder real-time applicability. To address this, we propose $\mathrm{T}^2$-VLM, a novel training-free, temporally consistent framework that generates accurate rewards through tracking the status changes in VLM-derived subgoals. Specifically, our method first queries the VLM to establish spatially aware subgoals and an initial completion estimate before each round of interaction. We then employ a Bayesian tracking algorithm to update the goal completion status dynamically, using subgoal hidden states to generate structured rewards for reinforcement learning (RL) agents. This approach enhances long-horizon decision-making and improves failure recovery capabilities with RL. Extensive experiments indicate that $\mathrm{T}^2$-VLM achieves state-of-the-art performance in two robot manipulation benchmarks, demonstrating superior reward accuracy with reduced computation consumption. We believe our approach not only advances reward generation techniques but also contributes to the broader field of embodied AI. Project website: https://t2-vlm.github.io/.
SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong(香港大学港大剑客基金会数据科学研究所);Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电气与电子工程系)
DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Carnegie Mellon University(卡内基梅隆大学);Shanghai Jiao Tong University(上海交通大学);Amazon AWS AI(亚马逊AWS人工智能)
TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models
School of Intelligent Science and Technology, Nanjing University, China(南京大学智能科学与技术学院);National Key Laboratory for Novel Software Technology, School of Artificial Intelligence, Nanjing University, China(南京大学新型软件技术国家实验室,人工智能学院);School of Artificial Intelligence, Nanjing University, China(南京大学人工智能学院);MACS Lab, University of Washington(华盛顿大学MACS实验室)
Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所 多模态人工智能系统国家重点实验室);School of Artificial Intelligence, University of the Chinese Academy of Sciences(中国科学院大学 人工智能学院);AGIBOT
MA-ROESL: Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos
College of Information Science and Technology, Eastern Institute of Technology(信息科学与技术学院,东部技术学院);School of Computer Science and Technology, The University of Science and Technology of China(计算机科学与技术学院,中国科学技术大学);Department of Computing, The Hong Kong Polytechnic University(计算学院,香港理工大学)