arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14253cs.CVcs.GR
收藏

MVD$^2$:面向多视图扩散的高效多视图三维重建

MVD$^2$: Efficient Multiview 3D Reconstruction for Multiview Diffusion

Tsinghua University(清华大学);Microsoft Research Asia(微软亚洲研究院)

Xin-Yang Zheng, Hao Pan, Yu-Xiao Guo, Xin Tong, Yang Liu

AI总结:

本文提出MVD$^2$,一种面向多视图扩散图像的高效三维重建方法,通过将图像特征聚合为三维特征体并解码为网格,结合视图相关训练方案,实现一秒内从多视图图像鲁棒生成高质量三维模型。

摘要:

作为一种有前景的三维生成技术,多视图扩散(MVD)因其在泛化能力、质量和效率方面的优势而受到广泛关注。通过使用三维数据微调预训练的大规模图像扩散模型,MVD方法首先基于图像或文本提示生成三维物体的多个视图,然后利用多视图三维重建恢复三维形状。然而,生成图像中的稀疏视图和不一致细节使三维重建面临挑战。我们提出MVD$^2$,一种面向多视图扩散(MVD)图像的高效三维重建方法。MVD$^2$通过投影和卷积将图像特征聚合到三维特征体中,然后将体特征解码为三维网格。我们使用三维形状集合以及由三维形状渲染视图提示生成的MVD图像来训练MVD$^2$。为解决生成的多视图图像与三维形状真值视图之间的差异,我们设计了一种简单而高效的视图相关训练方案。MVD$^2$提升了MVD的三维生成质量,并且速度快、对多种MVD方法具有鲁棒性。训练完成后,它能够在一秒内从多视图图像高效解码出三维网格。我们使用Zero-123++和ObjectVerse-LVIS三维数据集训练MVD$^2$,并展示了其在从不同MVD方法生成的多视图图像中生成三维模型方面的优越性能,同时使用合成图像和真实图像作为提示。

英文摘要:

As a promising 3D generation technique, multiview diffusion (MVD) has received a lot of attention due to its advantages in terms of generalizability, quality, and efficiency. By finetuning pretrained large image diffusion models with 3D data, the MVD methods first generate multiple views of a 3D object based on an image or text prompt and then reconstruct 3D shapes with multiview 3D reconstruction. However, the sparse views and inconsistent details in the generated images make 3D reconstruction challenging. We present MVD$^2$, an efficient 3D reconstruction method for multiview diffusion (MVD) images. MVD$^2$ aggregates image features into a 3D feature volume by projection and convolution and then decodes volumetric features into a 3D mesh. We train MVD$^2$ with 3D shape collections and MVD images prompted by rendered views of 3D shapes. To address the discrepancy between the generated multiview images and ground-truth views of the 3D shapes, we design a simple-yet-efficient view-dependent training scheme. MVD$^2$ improves the 3D generation quality of MVD and is fast and robust to various MVD methods. After training, it can efficiently decode 3D meshes from multiview images within one second. We train MVD$^2$ with Zero-123++ and ObjectVerse-LVIS 3D dataset and demonstrate its superior performance in generating 3D models from multiview images generated by different MVD methods, using both synthetic and real images as prompts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑