FashionEngine：通过多模态控制实现交互式 3D 人体生成与编辑
FashionEngine: Interactive 3D Human Generation and Editing via Multimodal Controls
AI总结：
本文提出 FashionEngine，利用预训练 3D 人体扩散模型、共享多模态 UV 空间与对齐采样器，实现文本、图像、草图等控制下的交互式 3D 数字人生成、编辑与虚拟试穿。
摘要：
我们提出 FashionEngine，这是一个交互式 3D 人体生成与编辑系统，可通过自然语言、视觉感知和手绘草图等用户友好的多模态控制创建 3D 数字人。FashionEngine 通过三个关键组件实现 3D 人体制作自动化：1）预训练 3D 人体扩散模型，从 2D 图像训练数据中学习在语义 UV 潜空间中对 3D 人体建模，为多样化生成和编辑任务提供强先验。2）Multimodality-UV Space 在规范的 UV 对齐空间中编码人体服装的纹理外观、形状拓扑和文本语义，将用户多模态输入与隐式 UV 潜空间忠实地对齐，以实现可控的 3D 人体编辑。该多模态-UV 空间在文本、图像和草图等不同用户输入之间共享，从而支持各种联合多模态编辑任务。3）Multimodality-UV Aligned Sampler 学习从扩散先验中采样高质量且多样化的 3D 人体。大量实验验证了 FashionEngine 在条件生成/编辑任务上的最先进性能。此外，我们为 FashionEngine 提供了交互式用户界面，在统一框架中支持条件与无条件生成任务，以及姿势/视角/形状控制、文本、图像和草图驱动的 3D 人体编辑和 3D 虚拟试穿等编辑任务。项目页面位于：https://taohuumd.github.io/projects/FashionEngine。
英文摘要：
We present FashionEngine, an interactive 3D human generation and editing system that creates 3D digital humans via user-friendly multimodal controls such as natural languages, visual perceptions, and hand-drawing sketches. FashionEngine automates the 3D human production with three key components: 1) A pre-trained 3D human diffusion model that learns to model 3D humans in a semantic UV latent space from 2D image training data, which provides strong priors for diverse generation and editing tasks. 2) Multimodality-UV Space encoding the texture appearance, shape topology, and textual semantics of human clothing in a canonical UV-aligned space, which faithfully aligns the user multimodal inputs with the implicit UV latent space for controllable 3D human editing. The multimodality-UV space is shared across different user inputs, such as texts, images, and sketches, which enables various joint multimodal editing tasks. 3) Multimodality-UV Aligned Sampler learns to sample high-quality and diverse 3D humans from the diffusion prior. Extensive experiments validate FashionEngine's state-of-the-art performance for conditional generation/editing tasks. In addition, we present an interactive user interface for our FashionEngine that enables both conditional and unconditional generation tasks, and editing tasks including pose/view/shape control, text-, image-, and sketch-driven 3D human editing and 3D virtual try-on, in a unified framework. Our project page is at: https://taohuumd.github.io/projects/FashionEngine.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
