arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.13976cs.CV
收藏

PlacidDreamer:推进文本到3D生成的和谐性

PlacidDreamer: Advancing Harmony in Text-to-3D Generation

Tsinghua University(清华大学);Kuaishou Technology(快手科技)

Shuo Huang, Shikun Sun, Zixuan Wang, Xiaoyu Qin, Yanmin Xiong, Yuan Zhang, Pengfei Wan, Di Zhang, Jia Jia

AI总结:

PlacidDreamer通过统一多视图扩散模型和平衡分数蒸馏算法,解决文本到3D生成中的方向冲突与过饱和问题,实现高质量3D资产生成。

摘要:

近年来,文本到3D生成引起了广泛关注,并带来了显著的性能提升。以往的方法利用端到端的3D生成模型来初始化3D高斯,利用多视图扩散模型来强制多视图一致性,并利用文本到图像扩散模型通过分数蒸馏算法来细化细节。然而,这些方法存在两个局限性。首先,由于不同模型旨在生成多样化的3D资产,它们在生成方向上存在冲突。其次,分数蒸馏中的过饱和问题尚未得到深入研究和解决。为了解决这些局限性,我们提出了PlacidDreamer,一个文本到3D框架,它通过单一的多视图扩散模型来协调初始化、多视图生成和文本条件生成,同时采用一种新颖的分数蒸馏算法来实现平衡的饱和度。为了统一生成方向,我们引入了Latent-Plane模块,这是一个易于训练的插件扩展,使多视图扩散模型能够为初始化提供快速的几何重建,并增强多视图图像以个性化文本到图像扩散模型。为了解决过饱和问题,我们提出将分数蒸馏视为一个多目标优化问题,并引入了平衡分数蒸馏算法,该算法提供了帕累托最优解,既能实现丰富的细节,又能实现平衡的饱和度。大量实验验证了我们的PlacidDreamer的卓越能力。代码可在\url{https://github.com/HansenHuang0823/PlacidDreamer}获取。

英文摘要:

Recently, text-to-3D generation has attracted significant attention, resulting in notable performance enhancements. Previous methods utilize end-to-end 3D generation models to initialize 3D Gaussians, multi-view diffusion models to enforce multi-view consistency, and text-to-image diffusion models to refine details with score distillation algorithms. However, these methods exhibit two limitations. Firstly, they encounter conflicts in generation directions since different models aim to produce diverse 3D assets. Secondly, the issue of over-saturation in score distillation has not been thoroughly investigated and solved. To address these limitations, we propose PlacidDreamer, a text-to-3D framework that harmonizes initialization, multi-view generation, and text-conditioned generation with a single multi-view diffusion model, while simultaneously employing a novel score distillation algorithm to achieve balanced saturation. To unify the generation direction, we introduce the Latent-Plane module, a training-friendly plug-in extension that enables multi-view diffusion models to provide fast geometry reconstruction for initialization and enhanced multi-view images to personalize the text-to-image diffusion model. To address the over-saturation problem, we propose to view score distillation as a multi-objective optimization problem and introduce the Balanced Score Distillation algorithm, which offers a Pareto Optimal solution that achieves both rich details and balanced saturation. Extensive experiments validate the outstanding capabilities of our PlacidDreamer. The code is available at \url{https://github.com/HansenHuang0823/PlacidDreamer}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑