arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.09878cs.CV
收藏

MVISTA-4D: 用于机器人操作的一致性视图4D世界模型与测试时动作推理

MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation

MMLab, The Chinese University of Hong Kong, Hong Kong SAR(香港理工大学MMLab,香港中文大学,香港特别行政区);The Hong Kong University of Science(香港理工大学);The University of Hong Kong(香港大学);Tsinghua University(清华大学)

Jiaxu Wang, Yicheng Jiang, Tianlun He, Jingkai Sun, Qiang Zhang, Jiahang Cao, Zesen Gan, Mingyuan Sun, Qiming Shao, Xiangyu Yue

AI总结:

提出一种基于世界模型的4D场景生成方法,通过多视图RGBD预测和测试时动作优化,实现几何一致的4D动态预测与机器人操作。

摘要:

基于世界模型的“想象-然后行动”范式成为机器人操作的一种有前景的方法,但现有方法通常仅支持纯图像预测或部分3D几何推理,限制了其预测完整4D场景动态的能力。本文提出了一种新颖的具身4D世界模型,能够实现几何一致、任意视图的RGBD生成:仅以单视图RGBD观测作为输入,模型想象其余视角,然后通过反投影和融合构建跨时间的更完整3D结构。为了高效学习多视图、跨模态生成,我们明确设计了跨视图和跨模态特征融合,共同促进RGB与深度之间的一致性,并强制视图间的几何对齐。除了预测,将生成的未来转换为动作通常由逆动力学处理,但这是病态的,因为多个动作可以解释相同的状态转换。我们通过一种测试时动作优化策略来解决这个问题,该策略通过生成模型反向传播以推断与预测未来最佳匹配的轨迹级潜在变量,以及一个残差逆动力学模型,将该轨迹先验转换为精确的可执行动作。在三个数据集上的实验表明,该方法在4D场景生成和下游操作任务上均表现出色,消融实验为关键设计选择提供了实用见解。

英文摘要:

World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. This work proposes a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model imagines the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Experiments on three datasets demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑