arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.26694cs.ROcs.AIcs.CV
收藏

从视频先验构建统一的4D世界动作模型

Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising

Tsinghua University(清华大学);Xiaomi Robotics(小米机器人);Peking University(北京大学);CASIA

Jun Guo, Qiwei Li, Peiyan Li, Zilong Chen, Nan Sun, Yifei Su, Heyun Wang, Yuan Zhang, Xinghang Li, Huaping Liu

AI总结:

本文提出X-WAM,统一了实时机器人动作执行与高保真的4D世界合成,在单一框架中解决传统统一世界模型仅建模2D像素空间且无法平衡动作效率与世界建模质量的问题。

摘要:

我们提出X-WAM,一种统一的4D世界模型,将实时机器人动作执行与高保真的4D世界合成(视频+3D重建)统一在一个框架中,解决了传统统一世界模型(如UWM)仅建模2D像素空间且无法平衡动作效率和世界建模质量的关键限制。为了利用预训练视频扩散模型的强大视觉先验,X-WAM通过预测多视角RGB-D视频来想象未来世界,并通过轻量级的结构适应高效获取空间信息:将预训练扩散变压器的最后几个块复制到专用的深度预测分支中,用于未来空间信息的重建。此外,我们提出异步噪声采样(ANS)以联合优化生成质量和动作解码效率。ANS在推理过程中应用专门的异步去噪调度,通过更少的步骤快速解码动作以实现高效的实时执行,同时将全部步骤用于生成高保真的视频。与训练过程中完全解耦时间步不同,ANS从它们的联合分布采样以与推理分布对齐。在超过5800小时的机器人数据上预训练,X-WAM在RoboCasa和RoboTwin 2.0基准测试中分别达到79.2%和90.7%的平均成功率,同时在视觉和几何指标上均超越现有方法的高保真4D重建和生成。

英文摘要:

We propose X-WAM, a Unified 4D World Model that unifies real-time robotic action execution and high-fidelity 4D world synthesis (video + 3D reconstruction) in a single framework, addressing the critical limitations of prior unified world models (e.g., UWM) that only model 2D pixel-space and fail to balance action efficiency and world modeling quality. To leverage the strong visual priors of pretrained video diffusion models, X-WAM imagines the future world by predicting multi-view RGB-D videos, and obtains spatial information efficiently through a lightweight structural adaptation: replicating the final few blocks of the pretrained Diffusion Transformer into a dedicated depth prediction branch for the reconstruction of future spatial information. Moreover, we propose Asynchronous Noise Sampling (ANS) to jointly optimize generation quality and action decoding efficiency. ANS applies a specialized asynchronous denoising schedule during inference, which rapidly decodes actions with fewer steps to enable efficient real-time execution, while dedicating the full sequence of steps to generate high-fidelity video. Rather than entirely decoupling the timesteps during training, ANS samples from their joint distribution to align with the inference distribution. Pretrained on over 5,800 hours of robotic data, X-WAM achieves 79.2% and 90.7% average success rate on RoboCasa and RoboTwin 2.0 benchmarks, while producing high-fidelity 4D reconstruction and generation surpassing existing methods in both visual and geometric metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑