arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.15841cs.CV
收藏

2L3: 将有缺陷的生成2D图像提升为精准3D模型

2L3: Lifting Imperfect Generated 2D Images into Accurate 3D

Zhejiang Lab(之江实验室);Zhejiang University(浙江大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)

Yizheng Chen, Rengan Xie, Qi Ye, Sen Yang, Zixuan Xie, Tianxiao Chen, Rong Li, Yuchi Huo

AI总结:

针对单图生成多视图存在的光照不一致、几何错位、视图稀疏导致3D重建质量差的问题,提出2L3框架,结合本征分解、瞬态单目先验与视图增强融合,实现更精准的单图3D重建,性能显著优于现有方法。

摘要:

从单张图像重建3D物体是一个引人关注却颇具挑战的问题。一种颇具前景的解决方案是利用多视图(MV)3D重建,将生成的多视图图像融合为一致的3D物体。然而,生成的图像通常存在光照不一致、几何结构未对齐以及视图稀疏的问题,导致重建质量不佳。为解决这些问题,我们提出了一种新颖的3D重建框架,分别利用本征分解引导、瞬态单目先验引导和视图增强来应对上述三个问题。具体而言,我们首先利用本征分解从生成图像中解耦着色信息,以降低光照不一致的影响;随后,我们引入带有视图相关瞬态编码的单目先验,以增强重建的法向量;最后,我们设计了一种视图增强融合策略,该策略最小化生成稀疏视图中的像素级损失以及增强随机视图中的语义损失,从而得到视图一致的几何结构和精细的纹理。因此,我们的方法能够将预训练的MV图像生成器与基于神经网络的体积有符号距离函数(SDF)表示相结合,实现单图像到3D物体的重建。我们在多个数据集上评估了该框架,并在定量和定性评估中均展现出优异性能,标志着3D物体重建领域的一项重大进展。与最新的前沿方法Syncdreamer相比,我们将倒角距离误差降低了约36%,并将峰值信噪比(PSNR)提升了约30%。

英文摘要:

Reconstructing 3D objects from a single image is an intriguing but challenging problem. One promising solution is to utilize multi-view (MV) 3D reconstruction to fuse generated MV images into consistent 3D objects. However, the generated images usually suffer from inconsistent lighting, misaligned geometry, and sparse views, leading to poor reconstruction quality. To cope with these problems, we present a novel 3D reconstruction framework that leverages intrinsic decomposition guidance, transient-mono prior guidance, and view augmentation to cope with the three issues, respectively. Specifically, we first leverage to decouple the shading information from the generated images to reduce the impact of inconsistent lighting; then, we introduce mono prior with view-dependent transient encoding to enhance the reconstructed normal; and finally, we design a view augmentation fusion strategy that minimizes pixel-level loss in generated sparse views and semantic loss in augmented random views, resulting in view-consistent geometry and detailed textures. Our approach, therefore, enables the integration of a pre-trained MV image generator and a neural network-based volumetric signed distance function (SDF) representation for a single image to 3D object reconstruction. We evaluate our framework on various datasets and demonstrate its superior performance in both quantitative and qualitative assessments, signifying a significant advancement in 3D object reconstruction. Compared with the latest state-of-the-art method Syncdreamer~\cite{liu2023syncdreamer}, we reduce the Chamfer Distance error by about 36\% and improve PSNR by about 30\% .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑