arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.20752cs.RO
收藏

GaussianDream:用于机器人操作的前馈3D高斯世界模型

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tsinghua University(清华大学);Zhejiang University(浙江大学);Beihang University(北京航空航天大学);Carnegie Mellon University(卡内基梅隆大学);The University of Hong Kong(香港大学)

Zijian Zhang, Yuqing Jiang, Qian Cheng, Xiaofan Li, Si Liu, Ding Zhao, Ping Luo, Weitao Zhou, Haibao Yu

AI总结:

提出GaussianDream,一种前馈3D高斯世界模型插件,通过可学习查询编码当前帧3D空间结构和短期未来演化,在训练时用静态重建和未来预测头监督,推理时仅保留查询条件化动作生成,在多个机器人操作基准上达到最先进性能。

摘要:

视觉-语言-动作(VLA)策略通过将预训练的视觉-语言模型的语义先验迁移到动作生成,推进了语言条件机器人操作。然而,标准的动作模仿学习通常缺乏对显式3D空间信息、密集几何监督和未来环境演化的充分建模,而这些对于精确的机器人交互至关重要。为解决这一问题,我们提出 extbf{GaussianDream},一种前馈3D高斯世界模型插件。具体地,我们在编码器中引入可学习的GaussianDream查询,使模型能够捕捉当前帧的3D空间结构和短时域的未来演化。训练时,潜在的高斯Dream前缀由静态重建头和未来预测头处理,生成当前3D高斯场景状态和未来高斯演化状态。当前分支通过RGB渲染和深度进行监督,而未来分支使用未来RGB、深度和伪3D场景流信号。推理时,GaussianDream丢弃所有辅助头,仅保留学习到的前缀以条件化动作生成,无需测试时的高斯重建或未来预测。实验结果表明,GaussianDream在多个机器人操作基准上取得了最先进的性能,在LIBERO上达到 extbf{98.4\%},在RoboCasa Human-50上达到 extbf{54.8\%},在真实机器人任务上达到 extbf{50.0\%}。与现有的3D增强VLA方法相比,GaussianDream在实现高精度的同时,提供了比基于视频的世界模型方法更高的推理效率。

英文摘要:

Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors from pretrained vision-language models to action generation. However, standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution, all critical for precise robotic interaction. To address this, we propose \textbf{GaussianDream}, a feed-forward 3D Gaussian world-model plug-in. Specifically, we introduce learnable GaussianDream Queries in the encoder, enabling the model to capture current-frame 3D spatial structure and short-horizon future evolution. During training, the latent GaussianDream prefix is processed by a static reconstruction head and a future prediction head to produce current 3D Gaussian scene states and future Gaussian evolution states. The current branch is supervised by RGB rendering and depth, while the future branch uses future RGB, depth, and pseudo 3D scene-flow signals. During inference, GaussianDream discards all auxiliary heads and retains only the learned prefix to condition action generation, without test-time Gaussian reconstruction or future prediction. Experimental results demonstrate that GaussianDream achieves state-of-the-art performance across multiple robotic manipulation benchmarks, reaching \textbf{98.4\%} on LIBERO, \textbf{54.8\%} on RoboCasa Human-50, and \textbf{50.0\%} on real-robot tasks. Compared with existing 3D-enhanced VLA methods, GaussianDream achieves strong accuracy while providing higher inference efficiency than video-based world-model approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑