arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.01162cs.CV
收藏

Get3DHuman:利用像素对齐重建先验将StyleGAN-Human提升为3D生成模型

Get3DHuman: Lifting StyleGAN-Human into a 3D Generative Model using Pixel-aligned Reconstruction Priors

CUHKSZ(香港中文大学(深圳));Tencent AI Lab(腾讯人工智能实验室);Tencent America(腾讯美国)

Zhangyang Xiong, Di Kang, Derong Jin, Weikai Chen, Linchao Bao, Shuguang Cui, Xiaoguang Han

AI总结:

本研究提出Get3DHuman 3D人体生成框架,通过先验网络桥接StyleGAN-Human隐空间与像素对齐3D重建器的特征体,结合定制损失,仅用少量3D真实数据就提升了生成质量,性能优于现有方法且支持多种应用。

摘要:

高质量3D数字人的快速生成对从娱乐到专业领域的大量应用都十分重要。可微渲染的最新进展使得无需3D真实标签数据也能训练3D生成模型,但生成的3D人体在保真度和多样性方面仍有很大提升空间。本文提出Get3DHuman这一新型3D人体生成框架,仅需少量3D真实标签数据预算,就能显著提升生成结果的真实感与多样性。我们的核心观察是,3D生成器可从2D人体生成器与3D重建器学到的人体相关先验中获益。具体而言,我们通过专门设计的先验网络,将Get3DHuman的隐空间与StyleGAN-Human的隐空间桥接,输入隐码被映射到由像素对齐3D重建器张成的形状与纹理特征体中。先验网络的输出随后作为主生成器网络的监督信号。为确保有效训练,我们还针对生成的特征体与中间特征图提出了三种定制损失函数。大量实验表明,Get3DHuman的性能大幅优于其他现有最优方法,还能支持形状插值、形状重纹理以及通过隐空间反演实现的单视图重建等广泛应用。

英文摘要:

Fast generation of high-quality 3D digital humans is important to a vast number of applications ranging from entertainment to professional concerns. Recent advances in differentiable rendering have enabled the training of 3D generative models without requiring 3D ground truths. However, the quality of the generated 3D humans still has much room to improve in terms of both fidelity and diversity. In this paper, we present Get3DHuman, a novel 3D human framework that can significantly boost the realism and diversity of the generated outcomes by only using a limited budget of 3D ground-truth data. Our key observation is that the 3D generator can profit from human-related priors learned through 2D human generators and 3D reconstructors. Specifically, we bridge the latent space of Get3DHuman with that of StyleGAN-Human via a specially-designed prior network, where the input latent code is mapped to the shape and texture feature volumes spanned by the pixel-aligned 3D reconstructor. The outcomes of the prior network are then leveraged as the supervisory signals for the main generator network. To ensure effective training, we further propose three tailored losses applied to the generated feature volumes and the intermediate feature maps. Extensive experiments demonstrate that Get3DHuman greatly outperforms the other state-of-the-art approaches and can support a wide range of applications including shape interpolation, shape re-texturing, and single-view reconstruction through latent inversion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑