arXiv 2403.091402024-03-14cs.CV

Sculpt3D：基于稀疏3D先验的多视图一致文本到3D生成

Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior

Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））；Centre for Frontier AI Research, A*STAR（新加坡前沿人工智能研究院（新加坡科技研究局下属））

Cheng Chen, Xiaofeng Yang, Fan Yang, Chengzeng Feng, Zhoujie Fu, Chuan-Sheng Foo, Guosheng Lin, Fayao Liu

备注：: Accepted by CVPR 2024. Project Page: https://stellarcheng.github.io/Sculpt3D/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

Sculpt3D通过从参考对象注入稀疏3D先验，无需重训2D扩散模型，利用关键点监督和视图调制，实现高质量多视图一致的文本到3D生成。

摘要：

近期关于文本到3D生成的研究表明，仅使用2D扩散监督进行3D生成往往会产生外观不一致（例如，背面出现人脸）和形状不准确（例如，动物多出腿）的结果。现有方法主要通过使用从3D数据渲染的图像重新训练扩散模型来确保多视图一致性，但难以平衡2D生成质量与3D一致性。在本文中，我们提出了一种新框架Sculpt3D，该框架在不重新训练2D扩散模型的情况下，将检索到的参考对象的3D先验显式注入到当前流程中。具体而言，我们证明了通过稀疏光线采样方法的关键点监督可以保证高质量且多样化的3D几何形状。此外，为了确保不同视图的外观准确性，我们进一步将2D扩散模型的输出调制为模板视图的正确模式，而不改变生成对象的风格。这两种解耦的设计有效地利用了参考对象的3D信息来生成3D对象，同时保持了2D扩散模型的生成质量。大量实验表明，我们的方法可以大幅提高多视图一致性，同时保持保真度和多样性。我们的项目页面可在以下网址获取：https://stellarcheng.github.io/Sculpt3D/。

英文摘要：

Recent works on text-to-3d generation show that using only 2D diffusion supervision for 3D generation tends to produce results with inconsistent appearances (e.g., faces on the back view) and inaccurate shapes (e.g., animals with extra legs). Existing methods mainly address this issue by retraining diffusion models with images rendered from 3D data to ensure multi-view consistency while struggling to balance 2D generation quality with 3D consistency. In this paper, we present a new framework Sculpt3D that equips the current pipeline with explicit injection of 3D priors from retrieved reference objects without re-training the 2D diffusion model. Specifically, we demonstrate that high-quality and diverse 3D geometry can be guaranteed by keypoints supervision through a sparse ray sampling approach. Moreover, to ensure accurate appearances of different views, we further modulate the output of the 2D diffusion model to the correct patterns of the template views without altering the generated object's style. These two decoupled designs effectively harness 3D information from reference objects to generate 3D objects while preserving the generation quality of the 2D diffusion model. Extensive experiments show our method can largely improve the multi-view consistency while retaining fidelity and diversity. Our project page is available at: https://stellarcheng.github.io/Sculpt3D/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。