arXiv 2607.024312026-07-03cs.ROcs.AI

WorldSample：基于世界建模的闭环真实机器人强化学习

WorldSample: Closed-loop Real-robot RL with World Modelling

PINE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电气与电子工程学院PINE实验室）；Department of Electronic Engineering, Tsinghua University, Beijing, China（清华大学电子工程系）；School of Automation, Central South University, Changsha, China（中南大学自动化学院）；School of Automation, Beijing University of Posts and Telecommunications, Beijing, China（北京邮电大学自动化学院）

Yuquan Xue, Le Xu, Zeyi Liu, Zhenyu Wu, Zhengyi Gu, Xinyang Song, Bofang Jia, Ziwei Wang

备注：: 16 pages, 9 figures, conference paper

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出WorldSample框架，通过物理 rollout、世界模型生成与策略改进的闭环，结合策略节奏学习（PPL）调节训练，在接触性精密操作任务中成功率提升28%，训练步数减少59%。

摘要：

强化学习（RL）通过允许机器人在超出演示中观察到的状态之外进行试错交互，克服了模仿学习（IL）的演示覆盖局限性。然而，在真实机器人上部署RL仍然受到高交互成本的限制，因为每次物理 rollout 成本高昂，并且仅反映一条已实现的行动-结果路径。为了解决这一挑战，我们提出了WorldSample，一种物理基础的数据增强框架，用于真实机器人RL，该框架在物理 rollout、世界模型生成和策略改进之间形成了一个真实-合成闭环。基于真实 rollout，WorldSample通过后训练的世界模型生成高保真合成转移，大大降低了视觉幻觉。具体来说，WorldSample不是简单地将这些转移用作真实世界经验，而是引入了策略节奏学习（PPL），通过样本选择和调度来调节训练过程，平衡有用增强与价值高估，并减轻幻觉引起的噪声。在涉及接触密集和精密任务的机器人操作实验表明，与基线相比，WorldSample将策略成功率提高了28%，同时将训练步骤减少了59%。此外，与仅演示后训练相比，WorldSample将世界模型的视觉保真度在PSNR上提高了19.4dB，在SSIM上提高了0.47，验证了真实-合成循环对策略和世界模型性能的有效性。

英文摘要：

Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL) by allowing robots to improve through trial-and-error interaction beyond the states observed in demonstrations. However, deploying RL on real robots remains constrained by high interaction costs, since each physical rollout is costly and reflects only one realized action-outcome path. To address this challenge, we propose WorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement. Grounded on real rollouts, WorldSample generates high-fidelity synthetic transitions through a post-trained world model, which greatly lowers the visual hallucination. Specifically, rather than simply using these transitions as real-world experience, WorldSample introduces Policy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling, balancing useful augmentation against value overestimation and mitigating the hallucination-induced noise. Experiments on robot manipulation tasks involving contact-rich and precise tasks show that WorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines. Furthermore, WorldSample improves world model visual fidelity by 19.4dB in PSNR and 0.47 in SSIM over demonstration-only post-training, validating the effectiveness of the real-synthetic loop for both policy and world model performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。