TOPReward: 令牌概率作为机器人学中的隐式零样本奖励
TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
AI总结：
TOPReward通过利用预训练视频视觉-语言模型的令牌概率，提供高效的零样本奖励估计，显著提升机器人任务进度评估的性能和泛化能力。
摘要：
尽管视觉-语言-动作（VLA）模型在预训练中取得了快速进展，但在强化学习（RL）中的进步仍然受到现实环境中样本效率低和稀疏奖励的限制。开发具有泛化能力的过程奖励模型对于提供必要的细粒度反馈以弥合这一差距至关重要，但现有的时序价值函数往往难以泛化到其训练领域之外。我们引入了TOPReward，一种基于概率的时序价值函数，利用预训练视频视觉-语言模型（VLMs）的潜在世界知识来估计机器人任务进度。与以往方法不同，这些方法提示VLMs直接输出进度值，容易导致数值误表征，而TOPReward直接从VLM内部的令牌日志中提取任务进度。在130多个不同的现实任务和多个机器人平台（例如Franka、YAM、SO-100/101）上的零样本评估中，TOPReward在Qwen3-VL上实现了0.947的平均价值顺序相关性（VOC），显著优于状态-of-the-art GVL基线在相同开源模型上接近零的相关性。我们进一步证明，TOPReward是下游应用中的多功能工具，包括成功检测和奖励对齐的行为克隆。
英文摘要：
General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior. Yet obtaining such feedback at scale remains difficult, since existing approaches often rely on manual progress annotations, task-specific demonstrations, or reward models trained on curated robot datasets. We introduce TOPReward, a training-free progress reward method that probes pretrained Video-Language Models (VLMs) through their internal token probabilities rather than asking them to generate numerical progress values. Given a video prefix and a language instruction, TOPReward measures the model's likelihood that the instructed task has been completed, converting latent video-language understanding into a dense reward signal without task-specific reward-model training or manually annotated progress labels. We evaluate TOPReward on ManiRewardBench, our real-world manipulation benchmark spanning 130 unique tasks and four robot platforms, as well as on Open X-Embodiment datasets. Across these settings, TOPReward substantially outperforms prior training-free VLM reward methods on open-source models and is competitive with a trained reward-model baseline on progress-estimation metrics, while requiring no reward-model training. Additional analyses show that the reward is sensitive to the specified instruction and is not explained by time index alone. Finally, TOPReward supports downstream applications including success detection and offline reward-weighted behavior cloning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
