arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.21082cs.CV
收藏

视频扩散模型能否重建4D几何？

Can Video Diffusion Model Reconstruct 4D Geometry?

King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）

Jinjie Mai, Wenxuan Zhu, Haozhe Liu, Bing Li, Cheng Zheng, Jürgen Schmidhuber, Bernard Ghanem

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Sora3R框架，利用视频扩散模型的时空先验，通过两阶段流程直接从单目视频生成4D点图，无需外部模块，实现与最先进方法相当的动态4D重建性能。

摘要：

从单目视频重建动态3D场景（即4D几何）是一个重要且具有挑战性的问题。传统的基于多视图几何的方法往往难以处理动态运动，而最近的基于学习的方法要么需要专门的4D表示，要么需要复杂的优化。在本文中，我们提出了Sora3R，一个新颖的框架，利用大规模视频扩散模型的丰富时空先验，直接从随意视频中推断4D点图。Sora3R遵循两阶段流程：（1）我们从预训练的视频VAE中适配一个点图VAE，确保几何和视频潜在空间之间的兼容性；（2）我们在视频和点图联合潜在空间中微调扩散骨干网络，以生成每一帧的连贯4D点图。Sora3R以完全前馈方式运行，不需要外部模块（例如深度、光流或分割）或迭代全局对齐。大量实验表明，Sora3R可靠地恢复了相机姿态和详细的场景几何，在多种场景下的动态4D重建中达到了与最先进方法相当的性能。

英文摘要：

Reconstructing dynamic 3D scenes (i.e., 4D geometry) from monocular video is an important yet challenging problem. Conventional multiview geometry-based approaches often struggle with dynamic motion, whereas recent learning-based methods either require specialized 4D representation or sophisticated optimization. In this paper, we present Sora3R, a novel framework that taps into the rich spatiotemporal priors of large-scale video diffusion models to directly infer 4D pointmaps from casual videos. Sora3R follows a two-stage pipeline: (1) we adapt a pointmap VAE from a pretrained video VAE, ensuring compatibility between the geometry and video latent spaces; (2) we finetune a diffusion backbone in combined video and pointmap latent space to generate coherent 4D pointmaps for every frame. Sora3R operates in a fully feedforward manner, requiring no external modules (e.g., depth, optical flow, or segmentation) or iterative global alignment. Extensive experiments demonstrate that Sora3R reliably recovers both camera poses and detailed scene geometry, achieving performance on par with state-of-the-art methods for dynamic 4D reconstruction across diverse scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑