arXiv 2508.176002025-09-17cs.ROcs.AIcs.CVcs.LG

GWM：迈向用于机器人操作的可扩展高斯世界模型

GWM: Towards Scalable Gaussian World Models for Robotic Manipulation

Tsinghua University（清华大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）

Guanxing Lu, Baoxiong Jia, Puhao Li, Yixin Chen, Ziwei Wang, Yansong Tang, Siyuan Huang

备注：: Published at ICCV 2025. Project page: https://gaussian-world-model.github.io/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出用于机器人操作的高斯世界模型GWM，通过推断高斯基元传播重建未来状态，结合DiT与3D VAE实现细粒度场景重建，支持模仿学习与基于模型的强化学习，显著超越现有方法。

摘要：

由于现实世界交互的低效性，在学习型世界模型中训练机器人策略正成为趋势。现有的基于图像的世界模型和策略已展现出初步成功，但缺乏对三维世界一致的空间和物理理解所需的稳健几何信息，即使是在互联网规模视频源上预训练也是如此。为此，我们提出一种名为高斯世界模型（GWM）的新型世界模型分支，用于机器人操作，它通过推断机器人动作影响下高斯基元的传播来重建未来状态。其核心是一个结合了3D变分自编码器的潜在Diffusion Transformer (DiT)，能够利用Gaussian Splatting进行细粒度的场景级未来状态重建。GWM不仅能通过自监督未来预测训练增强模仿学习智能体的视觉表示，还可作为支持基于模型的强化学习的神经模拟器。仿真和真实世界实验表明，GWM能精确预测以多样机器人动作为条件的未来场景，并进一步用于训练策略，以显著优势超越现有最先进方法，展现了3D世界模型初步的数据扩展潜力。

英文摘要：

Training robot policies within a learned world model is trending due to the inefficiency of real-world interactions. The established image-based world models and policies have shown prior success, but lack robust geometric information that requires consistent spatial and physical understanding of the three-dimensional world, even pre-trained on internet-scale video sources. To this end, we propose a novel branch of world model named Gaussian World Model (GWM) for robotic manipulation, which reconstructs the future state by inferring the propagation of Gaussian primitives under the effect of robot actions. At its core is a latent Diffusion Transformer (DiT) combined with a 3D variational autoencoder, enabling fine-grained scene-level future state reconstruction with Gaussian Splatting. GWM can not only enhance the visual representation for imitation learning agent by self-supervised future prediction training, but can serve as a neural simulator that supports model-based reinforcement learning. Both simulated and real-world experiments depict that GWM can precisely predict future scenes conditioned on diverse robot actions, and can be further utilized to train policies that outperform the state-of-the-art by impressive margins, showcasing the initial data scaling potential of 3D world model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。