arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.02596cs.CV
收藏

SweetDreamer:在2D Diffusion中对齐几何先验以实现一致的Text-to-3D

SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D

Hong Kong University of Science and Technology(香港科技大学);Light Illusions;South China University of Technology(华南理工大学);Tencent AI Lab(腾讯人工智能实验室)

Weiyu Li, Rui Chen, Xuelin Chen, Ping Tan

AI总结:

针对text-to-3D生成中的多视角不一致问题,提出通过微调2D diffusion模型生成视角特定坐标图以对齐2D几何先验与3D形状,在保留生成多样性的同时大幅提升一致性,达到85%以上的人类评估一致率。

摘要:

将预训练diffusion模型的2D结果提升到3D世界进行text-to-3D生成本质上是模糊的。2D diffusion模型仅学习视角无关的先验,因此在提升过程中缺乏3D知识,导致多视角不一致问题。我们发现该问题主要源于几何不一致,避免错位的几何结构能大幅缓解最终输出中的问题。因此,我们通过在提升过程中将diffusion模型中的2D几何先验与明确定义的3D形状对齐来提高一致性,解决了绝大部分问题。这是通过微调2D diffusion模型使其具有视角感知能力,并生成规范定向3D对象的视角特定坐标图来实现的。在我们的过程中,仅使用粗略的3D信息进行对齐。这种“粗略”对齐不仅解决了几何中的多视角不一致问题,还保留了2D diffusion模型生成3D数据集中未见过的细节丰富且多样化的高质量对象的能力。此外,我们的对齐几何先验(AGP)是通用的,可无缝集成到各种最先进的流水线中,在未见形状和视觉外观方面获得高泛化能力,同时极大缓解了多视角不一致问题。我们的方法通过人类评估实现了85%以上的一致性率,代表了新的最先进性能,而许多先前的方法约为30%。我们的项目页面是https://sweetdreamer3d.github.io/

英文摘要:

It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view-agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi-view inconsistency problem. We find that this problem primarily stems from geometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem in the final outputs. Therefore, we improve the consistency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-aware and to produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This "coarse" alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality objects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the-art pipelines, obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem. Our method represents a new state-of-the-art performance with an 85+% consistency rate by human evaluation, while many previous methods are around 30%. Our project page is https://sweetdreamer3d.github.io/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑