arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2402.10259cs.CVcs.GR

GaussianObject:基于Gaussian Splatting从四个视角实现高质量3D物体重建

GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting

Huawei Inc.(华为公司);University of Toronto(多伦多大学)

Chen Yang, Sikuang Li, Jiemin Fang, Ruofan Liang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian

AI总结:

针对稀疏视角下3D物体重建信息缺失和难以建立多视角一致性的挑战,提出GaussianObject框架,结合结构先验与扩散模型修复高斯表示,仅需4张图像即可实现高质量渲染并显著超越SOTA。

摘要:

从高度稀疏视角重建和渲染3D物体对于推动3D视觉技术的应用和提升用户体验至关重要。然而,稀疏视角图像包含的3D信息非常有限,导致两个重大挑战:1)由于用于匹配的图像太少,难以建立多视角一致性;2)由于视角覆盖不足,导致物体信息部分缺失或高度压缩。为了应对这些挑战,我们提出了GaussianObject,这是一个使用Gaussian Splatting表示和渲染3D物体的框架,仅需4张输入图像即可实现高渲染质量。我们首先引入visual hull和floater elimination技术,将结构先验显式注入初始优化过程以帮助建立多视角一致性,从而生成粗略的3D高斯表示。然后,我们基于扩散模型构建高斯修复模型来补充缺失的物体信息,其中高斯被进一步细化。我们设计了自生成策略来获取图像对,用于训练修复模型。我们进一步设计了无需COLMAP的变体,不需要预先给定准确的相机位姿,实现了具有竞争力的质量并促进了更广泛的应用。GaussianObject在MipNeRF360、OmniObject3D、OpenIllumination等数据集及我们收集的无位姿图像上进行了评估,仅从四个视角便取得了优越性能,显著超越了先前的SOTA方法。

英文摘要:

Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination, which explicitly inject structure priors into the initial optimization process to help build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. We further design a COLMAP-free variant, where pre-given accurate camera poses are not required, which achieves competitive quality and facilitates wider applications. GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, OpenIllumination, and our-collected unposed images, achieving superior performance from only four views and significantly outperforming previous SOTA methods. Our demo is available at https://gaussianobject.github.io/, and the code has been released at https://github.com/GaussianObject/GaussianObject.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑