RoboReward: 通用视觉-语言奖励模型用于机器人学
RoboReward: General-Purpose Vision-Language Reward Models for Robotics
AI总结：
RoboReward通过构建机器人奖励数据集和基准，训练视觉-语言奖励模型，验证了其在机器人学习中的有效性，并展示了改进策略学习和缩小与人工奖励差距的成果。
摘要：
设计良好的奖励对于有效基于强化学习的策略改进至关重要。在现实世界中，获取此类奖励通常需要要么劳动密集型的人工标注，要么脆弱的手工制定目标。视觉-语言模型（VLMs）已显示出作为自动奖励模型的潜力，但它们在真实机器人任务上的有效性尚不明确。在本工作中，我们旨在通过引入（1）基于大规模真实机器人语料库的机器人奖励数据集和基准（基于Open X-Embodiment (OXE)和RoboArena），以及（2）在此数据集上训练的视觉-语言奖励模型（RoboReward 4B/8B）来弥合这一差距。因为OXE是成功导向的且缺乏失败示例，我们提出了一种负例数据增强管道，通过反事实重新标记成功的片段和时间截取来生成校准的负例和近似失败案例。利用这一框架，我们构建了一个涵盖多样任务和形态的大型训练和评估数据集，以测试最先进的VLMs是否能够可靠地为机器人学习提供奖励。我们对开源和专有VLMs的评估发现，没有任何模型在所有任务上都表现出色，突显了显著的改进空间。然后，我们训练了通用的4B和8B参数模型，这些模型在为短时间 horizon 机器人任务分配奖励方面优于许多更大的VLMs。最后，我们将8B模型部署到真实的机器人强化学习中，并发现其在Gemini Robotics-ER 1.5之上提高了策略学习，同时缩小了与人类提供的奖励RL训练之间的差距。我们发布了完整的数据集、训练的奖励模型和评估套件，以促进通用奖励模型在机器人学中的发展：https://crfm.stanford.edu/helm/robo-reward-bench（项目网站）.
英文摘要：
A well-designed reward is critical for effective reinforcement learning-based policy improvement. In real-world robotics, obtaining such rewards typically requires either labor-intensive human labeling or brittle, handcrafted objectives. Vision-language models (VLMs) have shown promise as automatic reward models, yet their effectiveness on real robot tasks is poorly understood. In this work, we aim to close this gap by introducing (1) RoboReward, a robotics reward dataset and benchmark built on large-scale real-robot corpora from Open X-Embodiment (OXE) and RoboArena, and (2) vision-language reward models trained on this dataset (RoboReward 4B/8B). Because OXE is success-heavy and lacks failure examples, we propose a negative examples data augmentation pipeline that generates calibrated negative and near-misses via counterfactual relabeling of successful episodes and temporal clipping to create partial-progress outcomes from the same videos. Using this framework, we build a large training and evaluation dataset spanning diverse tasks and embodiments to test whether state-of-the-art VLMs can reliably provide rewards for robot learning. Our evaluation of open and proprietary VLMs finds that no model excels across tasks, highlighting substantial room for improvement. We then train general-purpose 4B- and 8B-parameter models that outperform much larger VLMs in assigning rewards for short-horizon robotic tasks. Finally, we deploy the 8B model in real-robot reinforcement learning and find that it improves policy learning over Gemini Robotics-ER 1.5 while narrowing the gap to RL training with human-provided rewards. We release the full dataset, trained reward models, and evaluation suite on our website to advance the development of general-purpose reward models in robotics: https://crfm.stanford.edu/helm/robo-reward-bench (project website).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
