arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06508cs.RO
收藏

World-VLA-Loop: 视频世界模型与VLA策略的闭环学习

World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy

Show Lab, National University of Singapore（新加坡国立大学Show实验室）

Xiaokang Liu, Zechen Bai, Hai Ci, Kevin Yuchen Ma, Mike Zheng Shou

备注：
16 pages, 9 figures
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出World-VLA-Loop框架，通过状态感知视频世界模型联合预测未来帧和二元奖励，并采用协同进化范式迭代优化VLA策略，减少对真实环境交互的依赖。

摘要：

强化学习（RL）可以超越行为克隆，优化视觉-语言-动作（VLA）策略，但由于需要大量 rollout、重置、监督和安全风险，真实世界的RL仍然昂贵。基于动作条件的视频世界模型提供了在虚拟环境中训练的选项，但它们在精确的动作跟随方面表现不佳，尤其是在细微的接近成功失败情况下。此外，它们缺乏用于RL的原生奖励信号。基于不准确的视觉预测计算奖励仍然不可靠。我们引入了World-VLA-Loop，它围绕两个基础设计和一个更高级别的协同进化范式构建。我们首先策划了SANS，专门混合成功和接近成功的轨迹，以改善动作-结果对齐。然后，我们训练了一个状态感知视频世界模型，该模型从扩散潜变量中联合预测未来帧和二元奖励。它将奖励估计与生成器耦合，而不是单独模块，从而反过来有利于视觉预测。由于RL过程中VLA行为会发生变化，固定的模拟器可能与更新后的策略不对齐，因此World-VLA-Loop通过使用精炼的世界模型进行迭代VLA后训练，同时将每个改进策略的rollout反馈回来增强和微调世界模型，从而形成闭环。在仿真和真实机器人实验中，World-VLA-Loop显著提高了VLA性能，同时减少了对昂贵的物理交互的依赖。

英文摘要：

Reinforcement learning (RL) can refine Vision-Language-Action (VLA) policies beyond behavior cloning, but real-world RL remains expensive due to extensive rollouts, resets, supervision, and safety risks. Action-conditioned video world models offer an option to train in virtual environments, yet they exhibit imprecise action following, particularly on subtle near-success failures. Besides, they lack native reward signals for RL. Computing rewards based on inaccurate visual predictions remain unreliable. We introduce World-VLA-Loop, structured around two foundational designs and a higher-level co-evolving paradigm. We first curate SANS, dedicatedly mixing successful and near-success trajectories to improve action-outcome alignment. Then, we train a state-aware video world model that jointly predicts future frames and binary rewards from diffusion latents. It couples reward estimation to the generator rather than a separate module, and in turn, benefits visual prediction. Since VLA behavior shifts during RL, a fixed simulator can misalign with the updated policy, World-VLA-Loop therefore closes the loop by using the refined world model for iterative VLA post-training while feeding rollouts from each improved policy back to augment and fine-tune the world model. Across simulation and real-robot experiments, World-VLA-Loop substantially improves VLA performance while reducing reliance on costly physical interaction.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑