具有不确定性的机器人世界模型使基于模型的离线强化学习在真实机器人上有效
Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
AI总结：
本文提出了一种具有不确定性的机器人世界模型，结合MOPO-PPO框架，使基于模型的离线强化学习在真实机器人上有效，通过提高策略的鲁棒性并超越现有基线。
摘要：
强化学习（RL）在机器人领域取得了令人印象深刻的结果，但高性能的管道仍然高度任务特定，很少重用先前的数据。基于模型的离线RL（MBRL）通过从现有数据集中训练策略来提高数据效率，但长期滚动中会出现累积误差和分布偏移。尽管现有方法在受控模拟基准中表现出色，但将它们稳健地应用于现实世界中典型的嘈杂、有偏见和部分观察的数据集仍然具有挑战性。我们提出了一个原理性的管道，使基于模型的离线RL在物理机器人上有效。我们的RWM-U在自回归世界模型中扩展了信念不确定性估计，使具有不确定性有效传播的时序一致的多步滚动成为可能。我们结合RWM-U与MOPO-PPO，将不确定性惩罚的策略优化适应到稳定的、基于策略的PPO框架中，用于现实世界的控制。我们在模拟和真实四足和人形机器人上评估了我们的方法，在离线数据集中训练策略。所得到的策略在模型自由和不确定性不敏感的模型基线中表现一致优异，融合现实数据在模型学习中进一步产生稳健的策略，超过仅在模拟中训练的在线模型自由基线。
英文摘要：
Reinforcement Learning (RL) has achieved impressive results in robotics, yet high-performing pipelines remain highly task-specific, with little reuse of prior data. Offline Model-based RL (MBRL) offers greater data efficiency by training policies entirely from existing datasets, but suffers from compounding errors and distribution shift in long-horizon rollouts. Although existing methods have shown success in controlled simulation benchmarks, robustly applying them to the noisy, biased, and partially observed datasets typical of real-world robotics remains challenging. We present a principled pipeline for making offline MBRL effective on physical robots. Our RWM-U extends autoregressive world models with epistemic uncertainty estimation, enabling temporally consistent multi-step rollouts with uncertainty effectively propagated over long horizons. We combine RWM-U with MOPO-PPO, which adapts uncertainty-penalized policy optimization to the stable, on-policy PPO framework for real-world control. We evaluate our approach on diverse manipulation and locomotion tasks in simulation and on real quadruped and humanoid, training policies entirely from offline datasets. The resulting policies consistently outperform model-free and uncertainty-unaware model-based baselines, and fusing real-world data in model learning further yields robust policies that surpass online model-free baselines trained solely in simulation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
