arXiv 2509.190802026-03-19cs.ROcs.AI

World4RL: 基于扩散世界模型的强化学习政策精修框架用于机器人操作

World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation

The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）；Beijing Zhongke Huiling Robot Technology Co（北京中科创联机器人技术有限公司）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）

Zhennan Jiang, Kai Liu, Yuxin Qin, Shuai Tian, Yupeng Zheng, Mingcai Zhou, Chao Yu, Haoran Li, Dongbin Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

World4RL通过扩散世界模型提升机器人操作政策的精修效果，采用高保真模拟环境进行端到端政策优化，优于模仿学习及其他基线方法。

摘要：

机器人操作策略通常通过模仿学习初始化，但其性能受限于专家数据的稀缺性和覆盖范围狭窄。强化学习可以精修策略以缓解这一限制，但真实机器人训练成本高且不安全，而仿真训练则面临仿真到现实的差距。生成模型的最新进展展示了在现实世界模拟中的强大能力，尤其是扩散模型在生成任务中表现突出。本文提出World4RL框架，利用基于扩散的世界模型作为高保真仿真器，在想象环境中完全精修预训练策略。与以往主要用于规划的世界模型不同，我们的框架实现了端到端的策略优化。World4RL围绕两个原则设计：预训练一个能够捕捉多任务数据集多样动态的扩散世界模型，以及在冻结的世界模型中完全精修策略以避免在线现实互动。我们进一步设计了适用于机器人操作的双热动作编码方案，并采用扩散骨干网络以提高建模保真度。广泛的仿真和现实实验表明，World4RL提供了高保真环境建模，并实现了一致的策略精修，相较于模仿学习和其他基线方法，取得了显著更高的成功率。

英文摘要：

Robotic manipulation policies are commonly initialized through imitation learning, but their performance is limited by the scarcity and narrow coverage of expert data. Reinforcement learning can refine polices to alleviate this limitation, yet real-robot training is costly and unsafe, while training in simulators suffers from the sim-to-real gap. Recent advances in generative models have demonstrated remarkable capabilities in real-world simulation, with diffusion models in particular excelling at generation. This raises the question of how diffusion model-based world models can be combined to enhance pre-trained policies in robotic manipulation. In this work, we propose World4RL, a framework that employs diffusion-based world models as high-fidelity simulators to refine pre-trained policies entirely in imagined environments for robotic manipulation. Unlike prior works that primarily employ world models for planning, our framework enables direct end-to-end policy optimization. World4RL is designed around two principles: pre-training a diffusion world model that captures diverse dynamics on multi-task datasets and refining policies entirely within a frozen world model to avoid online real-world interactions. We further design a two-hot action encoding scheme tailored for robotic manipulation and adopt diffusion backbones to improve modeling fidelity. Extensive simulation and real-world experiments demonstrate that World4RL provides high-fidelity environment modeling and enables consistent policy refinement, yielding significantly higher success rates compared to imitation learning and other baselines.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。