SOLE-R1：视频语言推理作为机器人强化学习的唯一奖励
SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
AI总结：
提出SOLE-R1模型，通过视频语言时空推理生成密集任务进度估计作为唯一奖励信号，实现在无真实奖励、演示或任务特定调优下的零样本在线强化学习。
摘要：
视觉语言模型（VLM）在各种任务中展现出令人印象深刻的能力，这促使人们努力利用这些模型来监督机器人学习。然而，当在强化学习（RL）中用作评估器时，当今最强的模型在部分可观测性和分布偏移下常常失败，使得策略能够利用感知错误而非解决任务。我们提出SOLE-R1（自观察学习器），一种专门设计用于为在线RL提供唯一奖励信号的视频语言推理模型。仅给定原始视频观测和自然语言目标，SOLE-R1执行每时间步的时空思维链（CoT）推理，并生成可直接用作奖励的密集任务进度估计。为了训练SOLE-R1，我们开发了一个大规模视频轨迹和推理合成流水线，生成与连续进度监督对齐的时间基础CoT轨迹。这些数据与基础的空间和多帧时间推理相结合，并使用混合框架训练模型，该框架将监督微调与可验证奖励的RL相结合。在四个不同的仿真环境和真实机器人设置中，SOLE-R1实现了从随机初始化的零样本在线RL：机器人学习之前未见过的操作任务，无需真实奖励、成功指标、演示或任务特定调优。SOLE-R1在24个未见过的任务上成功，并显著优于强视觉语言奖励器，包括Robometer、RoboReward、ReWiND、GPT-5和Gemini-3-Pro，同时对奖励破解表现出明显更强的鲁棒性。我们在匿名页面发布所有模型、数据、代码和演示：https://philip-mit.github.io/sole-r1/
英文摘要：
Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: https://philip-mit.github.io/sole-r1/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
