arXiv 2410.212992024-10-31cs.CV

TV-3DG：通过视觉提示掌握文本到3D定制化生成

TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt

Harbin Institute of Technology（哈尔滨工业大学）；Space AI, Li Auto（小鹏汽车太空人工智能）；School of Software, Tsinghua University（清华大学软件学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）

Jiahui Yang, Donglin Di, Baorui Ma, Xun Yang, Yongjia Ma, Wenzhang Sun, Wei Chen, Jianxun Cui, Zhou Xue, Meng Wang, Yebin Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对SDS技术在文本到3D定制化生成中难以适配多条件输入、生成质量低的问题，提出CSM算法并结合视觉提示融合、语义-几何校准模块构建TV-3DG方法，实现稳定高质量的定制化3D生成。

摘要：

近年来，生成模型的发展极大拓展了文本到3D生成的能力。许多方法依赖分数蒸馏采样（Score Distillation Sampling，SDS）技术。然而，SDS在定制化生成任务中难以适配文本、视觉提示等多条件输入。为探究核心原因，我们将SDS分解为差值项和无分类器引导项。分析发现，核心问题源于差值项以及优化过程中的随机噪声添加，二者都会导致蒸馏过程偏离目标模态。为解决该问题，我们提出了一种新算法——分类器分数匹配（Classifier Score Matching，CSM），该算法移除了SDS中的差值项，并采用确定性噪声添加过程以降低优化过程中的噪声，在我们的定制化生成框架中有效克服了SDS生成质量低的局限。基于CSM，我们通过注意力融合机制与采样引导技术整合视觉提示信息，形成了视觉提示CSM（Visual Prompt CSM，VPCSM）算法。此外，我们引入了语义-几何校准（Semantic-Geometry Calibration，SGC）模块，通过优化文本信息整合来提升生成质量。我们将该方法命名为TV-3DG，大量实验证明其能够实现稳定、高质量的定制化3D生成。项目页面：https://yjhboy.github.io/TV-3DG

英文摘要：

In recent years, advancements in generative models have significantly expanded the capabilities of text-to-3D generation. Many approaches rely on Score Distillation Sampling (SDS) technology. However, SDS struggles to accommodate multi-condition inputs, such as text and visual prompts, in customized generation tasks. To explore the core reasons, we decompose SDS into a difference term and a classifier-free guidance term. Our analysis identifies the core issue as arising from the difference term and the random noise addition during the optimization process, both contributing to deviations from the target mode during distillation. To address this, we propose a novel algorithm, Classifier Score Matching (CSM), which removes the difference term in SDS and uses a deterministic noise addition process to reduce noise during optimization, effectively overcoming the low-quality limitations of SDS in our customized generation framework. Based on CSM, we integrate visual prompt information with an attention fusion mechanism and sampling guidance techniques, forming the Visual Prompt CSM (VPCSM) algorithm. Furthermore, we introduce a Semantic-Geometry Calibration (SGC) module to enhance quality through improved textual information integration. We present our approach as TV-3DG, with extensive experiments demonstrating its capability to achieve stable, high-quality, customized 3D generation. Project page: \url{https://yjhboy.github.io/TV-3DG}

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。