arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.20607cs.ROcs.LG
收藏

面向视觉-语言-动作模型的实用世界模型强化学习

Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models

National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China(新型软件技术国家实验室,南京大学,南京,中国);School of Artificial Intelligence, Nanjing University, Nanjing, China(人工智能学院,南京大学,南京,中国);Mila - Quebec AI Institute(魁北克AI研究所)

Zhilong Zhang, Haoxiang Ren, Yihao Sun, Yifei Sheng, Haonan Wang, Haoxin Lin, Zhichao Wu, Pierre-Luc Bacon, Yang Yu

AI总结:

本文提出VLA-MBPO框架,解决视觉-语言-动作模型在强化学习中的世界建模、多视角一致性及稀疏奖励下的误差累积问题,提升策略性能和样本效率。

摘要:

视觉-语言-动作(VLA)模型在机器人控制中表现出强大的泛化能力,但通过强化学习(RL)微调时受到现实世界交互成本高和安全风险的限制。在交互式世界模型中训练VLA模型可避免这些问题,但带来了像素级世界建模、多视角一致性和稀疏奖励下的误差累积等挑战。基于大多模态模型和模型驱动强化学习的最新进展,我们提出了VLA-MBPO框架,以解决这些在VLA微调中的问题。我们的方法有三个关键设计:(i)利用统一多模态模型(UMMs)进行数据高效的world modeling;(ii)交错视角解码机制以强制多视角一致性;(iii)分块级分支回滚以缓解误差累积。理论分析和跨模拟和现实任务的实验表明,VLA-MBPO显著提高了策略性能和样本效率,凸显其在现实机器人部署中的鲁棒性和可扩展性。

英文摘要:

Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL, we propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning. Our approach has three key design choices: (i) adapting unified multimodal models (UMMs) for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding. Theoretical analysis and experiments across simulation and real-world tasks demonstrate that VLA-MBPO significantly improves policy performance and sample efficiency, underscoring its robustness and scalability for real-world robotic deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑