arXiv 2412.059292025-03-12cs.CV

通过2D编辑实现文本到3D生成

Text-to-3D Generation by 2D Editing

University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Aarhus University（奥胡斯大学）

Haoran Li, Yuli Tian, Yonghui Wang, Yong Liao, Lin Wang, Yuyang Wang, Peng Yuan Zhou

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出GE3D方法，将SDS重新理解为2D图像编辑过程，通过结合加噪轨迹与文本引导去噪轨迹并对齐潜变量，利用多步去噪蒸馏多粒度信息，实现照片级真实感的文本到3D生成。

摘要：

从预训练的2D扩散模型中蒸馏3D表示对于游戏、电影和室内设计等3D创意应用至关重要。当前基于SDS的方法受限于从扩散模型中信息蒸馏效率低下，这阻碍了照片级真实感3D内容的创建。在本文中，我们首先重新评估SDS方法，通过分析其基本本质——作为一种基础的图像编辑过程，由于单步去噪质量较差，通常会导致过饱和、过度平滑、缺乏丰富内容和多样性。鉴于此，我们随后提出了一种名为“通过编辑实现3D生成”（GE3D）的新方法。GE3D的每次迭代都利用一个2D编辑框架，该框架结合了一条加噪轨迹以保留输入图像的信息，以及一条文本引导的去噪轨迹。我们通过对齐两条轨迹上的潜变量来优化该过程。这种方法充分利用预训练的扩散模型，通过多个去噪步骤蒸馏多粒度信息，从而生成照片级真实感的3D输出。理论和实验结果均证实了我们方法的有效性，它不仅推动了3D生成技术的发展，还在3D生成与2D编辑之间建立了新颖的联系。这有可能激发该领域的进一步研究。代码和演示已发布在https://jahnsonblack.github.io/GE3D/。

英文摘要：

Distilling 3D representations from pretrained 2D diffusion models is essential for 3D creative applications across gaming, film, and interior design. Current SDS-based methods are hindered by inefficient information distillation from diffusion models, which prevents the creation of photorealistic 3D contents. In this paper, we first reevaluate the SDS approach by analyzing its fundamental nature as a basic image editing process that commonly results in over-saturation, over-smoothing, lack of rich content and diversity due to the poor-quality single-step denoising. In light of this, we then propose a novel method called 3D Generation by Editing (GE3D). Each iteration of GE3D utilizes a 2D editing framework that combines a noising trajectory to preserve the information of the input image, alongside a text-guided denoising trajectory. We optimize the process by aligning the latents across both trajectories. This approach fully exploits pretrained diffusion models to distill multi-granularity information through multiple denoising steps, resulting in photorealistic 3D outputs. Both theoretical and experimental results confirm the effectiveness of our approach, which not only advances 3D generation technology but also establishes a novel connection between 3D generation and 2D editing. This could potentially inspire further research in the field. Code and demos are released at https://jahnsonblack.github.io/GE3D/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。