arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.09499cs.CV
收藏

V-DPM:基于动态点图的4D视频重建

V-DPM: 4D Video Reconstruction with Dynamic Point Maps

University of Oxford(牛津大学)

Edgar Sucar, Eldar Insafutdinov, Zihang Lai, Andrea Vedaldi

AI总结:

V-DPM通过动态点图实现4D视频重建,利用预训练模型和合成数据提升动态场景的3D和4D重建性能。

摘要:

强大的3D表示,如DUSt3R不变点图,能够编码3D形状和相机参数,显著推动了前馈3D重建的发展。尽管点图假设场景静态,动态点图(DPMs)通过额外表示场景运动扩展了这一概念。然而,现有的DPMs仅限于图像对,并且像DUSt3R一样,在涉及超过两个视角时需要通过优化进行后处理。我们认为将DPMs应用于视频更具实用性,并引入V-DPM来证明这一点。首先,我们展示了如何以最大化表示能力、促进神经预测并启用预训练模型重用的方式,为视频输入制定DPMs。其次,我们在VGGT上实现了这些想法,VGGT是一个近期强大的3D重建器。尽管VGGT是在静态场景上训练的,但我们显示,少量合成数据足以将其适应为有效的V-DPM预测器。我们的方法在动态场景的3D和4D重建中实现了最先进的性能。特别是,与最近的VGGT动态扩展如P3不同,DPMs不仅能恢复动态深度,还能恢复场景中每个点的完整3D运动。

英文摘要:

Powerful 3D representations such as DUSt3R invariant point maps, which encode 3D shape and camera parameters, have significantly advanced feed forward 3D reconstruction. While point maps assume static scenes, Dynamic Point Maps (DPMs) extend this concept to dynamic 3D content by additionally representing scene motion. However, existing DPMs are limited to image pairs and, like DUSt3R, require post processing via optimization when more than two views are involved. We argue that DPMs are more useful when applied to videos and introduce V-DPM to demonstrate this. First, we show how to formulate DPMs for video input in a way that maximizes representational power, facilitates neural prediction, and enables reuse of pretrained models. Second, we implement these ideas on top of VGGT, a recent and powerful 3D reconstructor. Although VGGT was trained on static scenes, we show that a modest amount of synthetic data is sufficient to adapt it into an effective V-DPM predictor. Our approach achieves state of the art performance in 3D and 4D reconstruction for dynamic scenes. In particular, unlike recent dynamic extensions of VGGT such as P3, DPMs recover not only dynamic depth but also the full 3D motion of every point in the scene.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑