arXiv 2411.052732025-08-06cs.ROcs.AIcs.LG

基于视觉语言模型反馈的真实世界离线强化学习

Real-World Offline Reinforcement Learning from Vision Language Model Feedback

Indian Institute of Technology, Kharagpur（印度理工学院，克哈格浦尔分校）；IIIS, Tsinghua University（清华大学人工智能研究所）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）

Sreyas Venkataraman, Yufei Wang, Ziyu Wang, Navin Sriram Ravie, Zackory Erickson, David Held

备注：: 7 pages. Accepted at the LangRob Workshop 2024 @ CoRL, 2024. Accepted at 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种利用视觉语言模型偏好反馈自动为离线数据集生成奖励标签的系统，结合离线强化学习（隐式Q学习）在真实穿衣和模拟操作任务中显著优于基线方法。

摘要：

离线强化学习能够从预先收集的次优数据集中学习策略，而无需在线交互。这使得它非常适合真实世界机器人和安全关键场景，在这些场景中，收集在线数据或专家演示既缓慢、昂贵又危险。然而，大多数现有的离线强化学习工作假设数据集已经标注了任务奖励，这一过程通常需要大量人力，尤其是在真实世界中难以确定真实状态的情况下。在本文中，我们基于先前的工作（特别是RL-VLM-F）进行扩展，提出了一种新颖的系统，该系统利用视觉语言模型的偏好反馈和任务文本描述，自动为离线数据集生成奖励标签。然后，我们的方法使用带有奖励标签的数据集，通过离线强化学习来学习策略。我们展示了该系统在复杂的真实世界机器人辅助穿衣任务中的适用性，在该任务中，我们首先使用视觉语言模型在次优离线数据集上学习奖励函数，然后利用学习到的奖励采用隐式Q学习来开发有效的穿衣策略。我们的方法在涉及刚性和可变形物体操作的模拟任务中也表现良好，并且显著优于行为克隆和逆强化学习等基线方法。总之，我们提出了一种新系统，能够从未标记的次优离线数据集中实现自动奖励标记和策略学习。

英文摘要：

Offline reinforcement learning can enable policy learning from pre-collected, sub-optimal datasets without online interactions. This makes it ideal for real-world robots and safety-critical scenarios, where collecting online data or expert demonstrations is slow, costly, and risky. However, most existing offline RL works assume the dataset is already labeled with the task rewards, a process that often requires significant human effort, especially when ground-truth states are hard to ascertain (e.g., in the real-world). In this paper, we build on prior work, specifically RL-VLM-F, and propose a novel system that automatically generates reward labels for offline datasets using preference feedback from a vision-language model and a text description of the task. Our method then learns a policy using offline RL with the reward-labeled dataset. We demonstrate the system's applicability to a complex real-world robot-assisted dressing task, where we first learn a reward function using a vision-language model on a sub-optimal offline dataset, and then we use the learned reward to employ Implicit Q learning to develop an effective dressing policy. Our method also performs well in simulation tasks involving the manipulation of rigid and deformable objects, and significantly outperform baselines such as behavior cloning and inverse RL. In summary, we propose a new system that enables automatic reward labeling and policy learning from unlabeled, sub-optimal offline datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。