发表机构
University of North Carolina at Chapel Hill; Carnegie Mellon University; Shanghai Jiao Tong University; Amazon AWS AI（北卡罗来纳大学教堂山分校; 卡内基梅隆大学; 上海交通大学; 亚马逊AWS人工智能）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对强化学习中缺乏可靠奖励模型的问题，提出DenseReward模型，通过自动生成失败数据合成逼真轨迹，从视觉和语言预测密集奖励分数，在模拟和现实操作中表现优异，还为下游任务提供指导并发布相关资源。
AI 中文摘要
强化学习有望改进机器人策略，但因缺乏可靠的视觉语言奖励模型而受限。存在两个关键挑战：大规模获取多样失败数据和获得超越稀疏轨迹级成功标签的细粒度奖励信号。我们引入DenseReward，通过开发自动失败数据生成管道，在无人工标注情况下合成逼真失败轨迹，覆盖多种失败模式。该模型从视觉观察和语言指令预测密集帧级奖励分数，实验表明其在模拟和现实操作的密集奖励预测中优于通用VLM和现有机器人奖励模型，还为下游模型预测控制和强化学习提供有效奖励指导，且发布了数据集、训练好的奖励模型和评估套件。
英文摘要
Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning. However, its practical adoption remains bottlenecked by the lack of reliable vision-language reward models that provide dense and informative feedback. Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels. Collecting failure trajectories typically requires laborious human effort, while pseudo-failures constructed by relabeling successful demonstrations fail to capture the diverse physical failure modes that arise during robot execution. Meanwhile, existing reward models often predict sparse binary or trajectory-level rewards, which provide limited guidance for efficient policy optimization. We introduce DenseReward, a dense robotic reward model that addresses both challenges. To train DenseReward, we develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. DenseReward predicts dense frame-level reward scores from visual observations and language instructions, enabling fine-grained estimation of task progress throughout an episode. Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. We further demonstrate that DenseReward provides effective reward guidance for downstream model predictive control and reinforcement learning. We release the dataset, trained reward models, and evaluation suite to support the development of failure-aware dense reward modeling for robot learning.
CommentsWebsite: https://dense-reward.github.io/
