arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.04728cs.CVcs.AI
收藏

Morphable Diffusion:面向单图像虚拟形象创建的3D一致性扩散模型

Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation

ETH Zürich(苏黎世联邦理工学院);University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心);University Hospital Balgrist(巴尔格里斯大学医院);University of Zürich(苏黎世大学)

Xiyi Chen, Marko Mihajlovic, Shaofei Wang, Sergey Prokudin, Siyu Tang

AI总结:

本文将3D可变形模型集成到多视图一致扩散框架中,首次实现从单张图像生成完全3D一致、可动画化且照片级真实感的人类虚拟形象,并在新视图和新表情合成任务上优于现有方法。

摘要:

生成式扩散模型的最新进展使得从单张输入图像或文本提示生成3D资产这一此前难以实现的能力成为可能。在本工作中,我们旨在提升这些模型在创建可控、照片级真实感人类虚拟形象任务中的质量和功能性。我们通过将3D可变形模型集成到最先进的多视图一致性扩散方法中来实现这一目标。我们证明,在生成流程中对铰接式3D模型进行精确条件化,能够提升基线模型在单图像新视图合成任务上的性能。更重要的是,这种集成促进了面部表情和身体姿态控制被无缝且准确地纳入生成过程。据我们所知,我们提出的框架是首个能够从一张未见主体的单张图像创建完全3D一致、可动画化且照片级真实感人类虚拟形象的扩散模型;大量定量和定性评估表明,在新视图合成和新表情合成任务上,我们的方法优于现有的最先进虚拟形象创建模型。本项目的代码已公开发布。

英文摘要:

Recent advances in generative diffusion models have enabled the previously unfeasible capability of generating 3D assets from a single input image or a text prompt. In this work, we aim to enhance the quality and functionality of these models for the task of creating controllable, photorealistic human avatars. We achieve this by integrating a 3D morphable model into the state-of-the-art multi-view-consistent diffusion approach. We demonstrate that accurate conditioning of a generative pipeline on the articulated 3D model enhances the baseline model performance on the task of novel view synthesis from a single image. More importantly, this integration facilitates a seamless and accurate incorporation of facial expression and body pose control into the generation process. To the best of our knowledge, our proposed framework is the first diffusion model to enable the creation of fully 3D-consistent, animatable, and photorealistic human avatars from a single image of an unseen subject; extensive quantitative and qualitative evaluations demonstrate the advantages of our approach over existing state-of-the-art avatar creation models on both novel view and novel expression synthesis tasks. The code for our project is publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑