arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.31377cs.ROcs.AI
收藏

面向强化学习的阶段转换密集奖励建模

Stage-Transition Dense Reward Modeling for Reinforcement Learning

Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Zerith Robotics

Yang Yang, Bingjie Chen, Zihan Wang, Yizhe Li, Guoping Pan, Yi Cheng, Houde Liu

AI总结:

提出阶段转换密集奖励(STDR)框架,利用专家视频学习任务阶段结构,提供阶段转换和阶段内进展两种密集奖励信号,结合OOD检测和抓取调节模块,提升机器人操作任务的样本效率和成功率。

摘要:

面向长时域机器人操作的强化学习常受稀疏和延迟奖励的限制,而手动设计密集塑形信号成本高且对环境与物体配置变化脆弱。本文提出阶段转换密集奖励(STDR),一种视觉奖励学习框架,将非结构化专家视频转化为逻辑上合理的密集奖励,用于从零训练强化学习智能体。STDR利用语义理解从演示中推断任务的阶段结构,并在在线训练期间提供两种互补的学习信号:(i)阶段转换反馈,提供目标导向的奖励;(ii)阶段内进展反馈,提供完成每个阶段的细粒度指导。此外,集成了分布外(OOD)检测机制和抓取调节模块以增强鲁棒性并防止奖励黑客。在MetaWorld、ManiSkill和Franka Kitchen上的14个操作任务实验表明,STDR在多个基线上持续提升样本效率和成功率,并在多个具有挑战性的任务上匹配或超越手工设计的密集奖励。真实机器人评估进一步表明,STDR在成功执行时分配稳定、与进展对齐的奖励,同时对失败产生适当低的奖励,表明其对视觉噪声的鲁棒性和跨设置的更好校准的奖励分配。

英文摘要:

Reinforcement learning for long-horizon robotic manipulation is often limited by sparse and delayed rewards, while manually designing dense shaping signals is costly and brittle to changes in environments and object configurations. This work proposes Stage-Transition Dense Reward (STDR), a visual reward-learning framework that converts unstructured expert videos into logically grounded dense rewards for training RL agents from scratch. STDR leverages semantic understanding to infer a task's stage structure from demonstrations, and delivers two complementary learning signals during online training: (i) stage-transition feedback that provides goal-directed reward, and (ii) within-stage progress feedback that supplies fine-grained guidance toward completing each stage. Furthermore, an out-of-distribution (OOD) detection mechanism and a grasping regulation module are integrated to enhance robustness and prevent reward hacking. Experiments on 14 manipulation tasks across MetaWorld, ManiSkill, and Franka Kitchen show that STDR consistently improves sample efficiency and success rates over multiple baselines, and matches or surpasses handcrafted dense rewards on several challenging tasks. Real-robot evaluations further indicate that STDR assigns stable, progress-aligned rewards on successful executions while producing appropriately low rewards for failures, suggesting robustness to visual noise and better-calibrated reward assignment across settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑