arXiv 2406.100002024-06-14cs.CV

OrientDream：通过显式方向控制简化文本到3D生成

OrientDream: Streamlining Text-to-3D Generation with Explicit Orientation Control

OPPO US Research Center（OPPO美国研究中心）；University of Southern California（南加州大学）；Nanyang Technological University（南洋理工大学）

Yuzhong Huang, Zhong Li, Zhang Chen, Zhiyuan Ren, Guosheng Lin, Fred Morstatter, Yi Xu

AI总结：

本文提出OrientDream框架，通过在2D扩散模型中引入显式相机方向条件并利用MVImgNet预训练，结合解耦反向传播加速NeRF优化，解决了Dreamfusion的多头Janus问题和优化缓慢问题，实现了高效且多视图一致的文本到3D生成。

摘要：

在不断发展的文本到3D技术领域中，Dreamfusion通过利用分数蒸馏采样（SDS）来优化诸如NeRF之类的隐式表示，展示了其能力。该过程通过蒸馏预训练的大规模文本到图像扩散模型来实现。然而，Dreamfusion面临保真度和效率方面的限制：它存在多头Janus问题，并且优化过程相对较慢。为了规避这些挑战，我们提出了OrientDream，这是一个以相机方向为条件的框架，旨在从文本提示中实现高效且多视图一致的3D生成。我们的策略强调在2D文本到图像扩散模块的预训练中实现显式的相机方向条件特征。该特征有效利用了MVImgNet这一广泛的外部多视图数据集，以完善和增强其功能。随后，我们利用预条件的2D图像作为优化随机初始化的隐式表示（NeRF）的基础。这一过程通过解耦的反向传播技术显著加速，允许在每个优化周期内对隐式参数进行多次更新。我们的实验表明，我们的方法不仅能够生成具有一致多视图属性的高质量NeRF模型，而且通过对比指标量化，其优化速度显著快于现有方法。

英文摘要：

In the evolving landscape of text-to-3D technology, Dreamfusion has showcased its proficiency by utilizing Score Distillation Sampling (SDS) to optimize implicit representations such as NeRF. This process is achieved through the distillation of pretrained large-scale text-to-image diffusion models. However, Dreamfusion encounters fidelity and efficiency constraints: it faces the multi-head Janus issue and exhibits a relatively slow optimization process. To circumvent these challenges, we introduce OrientDream, a camera orientation conditioned framework designed for efficient and multi-view consistent 3D generation from textual prompts. Our strategy emphasizes the implementation of an explicit camera orientation conditioned feature in the pre-training of a 2D text-to-image diffusion module. This feature effectively utilizes data from MVImgNet, an extensive external multi-view dataset, to refine and bolster its functionality. Subsequently, we utilize the pre-conditioned 2D images as a basis for optimizing a randomly initialized implicit representation (NeRF). This process is significantly expedited by a decoupled back-propagation technique, allowing for multiple updates of implicit parameters per optimization cycle. Our experiments reveal that our method not only produces high-quality NeRF models with consistent multi-view properties but also achieves an optimization speed significantly greater than existing methods, as quantified by comparative metrics.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。