arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.18532cs.ROcs.AIcs.LG
收藏

在生成3D世界中扩展机器人视觉语言动作的强化学习

Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worlds

Horizon Robotics(地平线机器人)

Andrew Choi, Xinjie Wang, Zhizhong Su, Wei Xu

AI总结:

本文提出利用生成3D世界模型来提升机器人视觉语言动作模型的泛化能力,通过生成多样化的交互场景,提高模拟到现实的迁移效果,并展示在真实世界中的性能提升。

摘要:

大规模视觉语言模型(VLM)通过强化学习(RL)训练表现出强大性能,激励了类似方法用于机器人视觉语言动作(VLA)模型的微调。许多最近的工作直接在现实世界中微调VLA,以避免仿真到现实的差距。虽然现实世界的RL可以规避仿真到现实的问题,但其本质上限制了VLA的泛化能力,因为在物理世界中扩展场景和物体多样性是难以实现的。这导致了将广泛预训练的模型转换为过拟合、场景特定策略的悖论性结果。通过仿真训练可以提供多样化的场景,但设计这些场景也是昂贵的。在本工作中,我们证明VLA可以在不牺牲泛化能力的情况下进行RL微调,并通过利用3D世界生成模型减少劳动量。使用这些模型结合语言驱动的场景设计师,我们生成了数百个多样化的交互场景,包含独特的物体和背景,使可扩展且高度并行的策略学习成为可能。从一个预训练的模仿基线开始,我们的方法将模拟成功率从9.7%提高到79.8%,同时在任务完成时间上实现了1.25倍的加速。我们进一步展示了通过生成的数字双胞胎质量和领域随机化实现的成功仿真到现实迁移,将现实世界成功率从21.7%提高到75%,并实现了1.13倍的加速。最后,我们进一步强调了利用3D世界生成模型提供的有效无限数据的好处,通过消融研究显示,增加场景多样性直接提高了零样本泛化能力。

英文摘要:

The strong performance of large vision-language models (VLMs) trained with reinforcement learning (RL) has motivated similar approaches for fine-tuning vision-language-action (VLA) models in robotics. Many recent works fine-tune VLAs directly in the real world to avoid addressing the sim-to-real gap. While real-world RL circumvents sim-to-real issues, it inherently limits the generality of the resulting VLA, as scaling scene and object diversity in the physical world is prohibitively difficult. This leads to the paradoxical outcome of transforming a broadly pretrained model into an overfitted, scene-specific policy. Training in simulation can instead provide access to diverse scenes, but designing those scenes is also costly. In this work, we show that VLAs can be RL fine-tuned across broad scene and object distributions and with reduced labor by leveraging 3D world generative models. Using these models together with a language-driven scene designer, we generate 100 diverse interactive scenes containing unique objects and backgrounds, enabling scalable and highly parallel policy learning. Starting from a pretrained imitation baseline, our approach increases simulation success from 9.7% up to 79.8% while achieving a 1.25$\times$ speedup in task completion time. We further demonstrate successful sim-to-real transfer enabled by the quality of the generated scenes together with domain randomization, improving real-world success from 21.7% to 75% and achieving a 1.13$\times$ speedup. Finally, we further highlight the benefits of leveraging the effectively unlimited data from 3D world generative models through an ablation study showing that increasing scene diversity directly improves zero-shot generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑