arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.11273cs.CV
收藏

BrightDreamer:用于快速文本到3D合成的通用3D高斯生成框架

BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesis

AI Thrust, HKUST(GZ)(人工智能推进组,香港科技大学(广州));NTU Singapore(新加坡南洋理工大学)

Lutao Jiang, Xu Zheng, Yuanhuiyi Lyu, Jiazhou Zhou, Lin Wang

AI总结:

本文提出BrightDreamer,一种端到端前馈文本到3D生成框架,通过TSD网络预测3D高斯中心位置,并利用TTG生成三平面表示估计其余属性,实现77毫秒快速生成且达每秒705帧渲染速度,显著优于现有方法。

摘要:

文本到3D合成近期通过Score Distillation Sampling (SDS)将文本到图像先验与3D表示方法(例如3D Gaussian Splatting (3D GS))相结合,取得了引人注目的进展。然而,现有方法面临的一个障碍是效率低下,即针对单个3D对象进行逐提示词优化。因此,从逐提示词优化向针对任何未见文本提示词的前馈生成进行范式转换势在必行,但这仍然具有挑战性。其中一个障碍是如何直接生成数以百万计的3D高斯集合来表示一个3D对象。本文提出了BrightDreamer,这是一种端到端的前馈方法,能够实现可泛化且快速(77 ms)的文本到3D生成。我们的核心思想是将生成过程表述为从具有预定义位置的锚点形状估计3D形变。为此,我们首先提出了一个Text-guided Shape Deformation (TSD)网络,用于预测形变后的形状及其新位置,用作3D高斯的中心(其中一个属性)。为了估计其他四个属性(即缩放、旋转、不透明度和SH),我们随后设计了一个新颖的Text-guided Triplane Generator (TTG),为3D对象生成三平面表示。每个高斯的中心使我们能够将空间特征转换为这四个属性。生成的3D高斯最终可以以每秒705帧的速度进行渲染。大量实验证明了我们的方法相对于现有方法的优越性。此外,BrightDreamer即使对于复杂的文本提示词也具备强大的语义理解能力。代码可在项目页面获取。

英文摘要:

Text-to-3D synthesis has recently seen intriguing advances by combining the text-to-image priors with 3D representation methods, e.g., 3D Gaussian Splatting (3D GS), via Score Distillation Sampling (SDS). However, a hurdle of existing methods is the low efficiency, per-prompt optimization for a single 3D object. Therefore, it is imperative for a paradigm shift from per-prompt optimization to feed-forward generation for any unseen text prompts, which yet remains challenging. An obstacle is how to directly generate a set of millions of 3D Gaussians to represent a 3D object. This paper presents BrightDreamer, an end-to-end feed-forward approach that can achieve generalizable and fast (77 ms) text-to-3D generation. Our key idea is to formulate the generation process as estimating the 3D deformation from an anchor shape with predefined positions. For this, we first propose a Text-guided Shape Deformation (TSD) network to predict the deformed shape and its new positions, used as the centers (one attribute) of 3D Gaussians. To estimate the other four attributes (i.e., scaling, rotation, opacity, and SH), we then design a novel Text-guided Triplane Generator (TTG) to generate a triplane representation for a 3D object. The center of each Gaussian enables us to transform the spatial feature into the four attributes. The generated 3D Gaussians can be finally rendered at 705 frames per second. Extensive experiments demonstrate the superiority of our method over existing methods. Also, BrightDreamer possesses a strong semantic understanding capability even for complex text prompts. The code is available in the project page.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑