持久的机器人世界模型：通过强化学习稳定多步 rollout
Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning
AI总结：
本文通过强化学习方法改进机器人世界模型，解决多步预测中误差累积问题，提出自回归rollout训练方案和多视角视觉反馈奖励，提升预测精度和稳定性，实现在DROID数据集上的state-of-the-art表现。
摘要：
基于动作的机器人世界模型通过给定机器人动作序列生成未来的视频帧，为难以用传统物理引擎模拟的任务提供了一种有前景的替代方案。然而，这些模型优化于短期预测，在自回归部署时会崩溃：每个预测的片段反馈作为下一部分的上下文，导致误差累积和视觉质量迅速下降。我们通过以下贡献来解决这一问题。首先，我们引入了一种强化学习（RL）后训练方案，该方案通过训练世界模型自身自回归的rollouts而非地面真实历史来训练世界模型。我们通过将最近的对比RL目标适应到扩散模型设置中，并展示其收敛保证精确延续。其次，我们设计了一种训练协议，从同一rollout状态生成并比较多个候选变量长度的未来，强化高保真度的预测优于低保真度的预测。第三，我们开发了高效的多视角视觉保真度奖励，结合了互补的感知度量标准，并在片段级别汇总，以实现密集且低方差的训练信号。第四，我们展示了我们的方法在DROID数据集上的rollout保真度达到了新的状态-of-the-art，优于所有基线（例如，外部摄像头上的LPIPS减少了14%，手腕摄像头上的SSIM提高了9.1%），在配对比较中赢得了98%的胜利，并在盲人研究中实现了80%的偏好率。
英文摘要：
Action-conditioned robot world models generate future video frames of the manipulated scene given a robot action sequence, offering a promising alternative for simulating tasks that are difficult to model with traditional physics engines. However, these models are optimized for short-term prediction and break down when deployed autoregressively: each predicted clip feeds back as context for the next, causing errors to compound and visual quality to rapidly degrade. We address this through the following contributions. First, we introduce a reinforcement learning (RL) post-training scheme that trains the world model on its own autoregressive rollouts rather than on ground-truth histories. We achieve this by adapting a recent contrastive RL objective for diffusion models to our setting and show that its convergence guarantees carry over exactly. Second, we design a training protocol that generates and compares multiple candidate variable-length futures from the same rollout state, reinforcing higher-fidelity predictions over lower-fidelity ones. Third, we develop efficient, multi-view visual fidelity rewards that combine complementary perceptual metrics across camera views and are aggregated at the clip level for dense, low-variance training signal. Fourth, we show that our approach establishes a new state-of-the-art for rollout fidelity on the DROID dataset, outperforming the strongest baseline on all metrics (e.g., LPIPS reduced by 14% on external cameras, SSIM improved by 9.1% on the wrist camera), winning 98% of paired comparisons, and achieving an 80% preference rate in a blind human study.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
