arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.02152cs.CV

GeneAvatar:从单张图像进行通用的表情感知体积化头部化身编辑

GeneAvatar: Generic Expression-Aware Volumetric Head Avatar Editing from a Single Image

Google(谷歌);Zhejiang University(浙江大学);ByteDance(字节跳动);ETH Zürich(苏黎世联邦理工学院)

Chong Bao, Yinda Zhang, Yuan Li, Xiyu Zhang, Bangbang Yang, Hujun Bao, Marc Pollefeys, Guofeng Zhang, Zhaopeng Cui

AI总结:

针对跨体积化表示的3D头部化身编辑难题,提出GeneAvatar,利用表情感知修改生成模型将单图2D编辑提升为一致3D修改场,并通过蒸馏、潜空间引导和损失重加权实现多表情多视角的高质量编辑。

摘要:

近来,我们见证了各类体积化表示在建模可动画头部化身方面的爆发式增长。然而,由于框架的多样性,目前尚无实用方法能够支持跨不同表示的3D头部化身编辑等高层应用。本文提出一种通用化身编辑方法,可普遍应用于各种由3DMM驱动的体积化头部化身。为实现这一目标,我们设计了一种新颖的表情感知修改生成模型,能够将单张图像上的2D编辑提升为一致的3D修改场。为确保生成式修改过程的有效性,我们开发了若干技术,包括用于从大规模头部化身模型和2D面部纹理编辑工具中汲取知识的表情依赖型修改蒸馏方案、用于增强模型收敛的隐式潜空间引导,以及用于细粒度纹理反演的基于分割的损失重加权策略。大量实验表明,我们的方法在多种表情和视角下均能提供高质量且一致的结果。项目页面:https://zju3dv.github.io/geneavatar/

英文摘要:

Recently, we have witnessed the explosive growth of various volumetric representations in modeling animatable head avatars. However, due to the diversity of frameworks, there is no practical method to support high-level applications like 3D head avatar editing across different representations. In this paper, we propose a generic avatar editing approach that can be universally applied to various 3DMM driving volumetric head avatars. To achieve this goal, we design a novel expression-aware modification generative model, which enables lift 2D editing from a single image to a consistent 3D modification field. To ensure the effectiveness of the generative modification process, we develop several techniques, including an expression-dependent modification distillation scheme to draw knowledge from the large-scale head avatar model and 2D facial texture editing tools, implicit latent space guidance to enhance model convergence, and a segmentation-based loss reweight strategy for fine-grained texture inversion. Extensive experiments demonstrate that our method delivers high-quality and consistent results across multiple expression and viewpoints. Project page: https://zju3dv.github.io/geneavatar/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑