arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.02454cs.ROcs.AI
收藏

World-Gymnast: 在世界模型中用强化学习训练机器人

World-Gymnast: Training Robots with Reinforcement Learning in a World Model

New York University(纽约大学);University of California, Berkeley(加州大学伯克利分校)

Ansh Kumar Sharma, Yixiang Sun, Ninghao Lu, Yunzhe Zhang, Jiarao Liu, Sherry Yang

AI总结:

World-Gymnast通过在世界模型中进行强化学习微调,有效提升真实机器人性能,比监督学习和软件模拟更具优势。

摘要:

机器人通过与物理世界交互进行学习,本质上受到物理交互成本的限制。两种替代方法,即从专家演示中进行监督微调(SFT)和在软件模拟器中进行强化学习(RL),受到可用专家数据量和模拟到现实差距的限制。随着最近世界模型从真实世界视频-动作数据中学习的出现,我们提出了一个问题:在世界模型中训练策略是否比监督学习或软件模拟更有效,以实现更好的真实机器人性能。我们提出了World-Gymnast,它通过在动作条件化的视频世界模型中 rollout 策略,并用视觉语言模型(VLM)对 rollout 进行奖励,从而对视觉语言动作(VLA)策略进行强化学习微调。在Bridge机器人平台上,World-Gymnast比SFT高出18倍,比软件模拟器高出2倍。更重要的是,World-Gymnast展示了在世界模型中进行强化学习的有趣能力,包括在世界模型中训练多样化的语言指令和新的场景,测试时在新场景中进行训练,以及在线迭代的世界模型和策略改进。我们的结果表明,学习世界模型并在云中训练机器人策略可能是连接在演示中工作的机器人和在任何人家庭中工作的机器人的关键。

英文摘要:

Robot learning from interacting with the physical world is fundamentally bottlenecked by the cost of physical interaction. The two alternatives, supervised finetuning (SFT) from expert demonstrations and reinforcement learning (RL) in a software-based simulator, are limited by the amount of expert data available and the sim-to-real gap for manipulation. With the recent emergence of world models learned from real-world video-action data, we ask the question of whether training a policy in a world model can be more effective than supervised learning or software simulation in achieving better real-robot performance. We propose World-Gymnast, which performs RL finetuning of a vision-language-action (VLA) policy by rolling out the policy in an action-conditioned video world model and rewarding the rollouts with a vision-language model (VLM). On the Bridge robot setup, World-Gymnast outperforms SFT by as much as 18x and outperforms software simulator by as much as 2x. More importantly, World-Gymnast demonstrates intriguing capabilities of RL with a world model, including training on diverse language instructions and novel scenes from the world model, test-time training in a novel scene, and online iterative world model and policy improvement. Our results suggest learning a world model and training robot policies in the cloud could be the key to bridging the gap between robots that work in demonstrations and robots that can work in anyone's household.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑