Facial expression retargeting from humans to virtual characters is a useful technique in computer graphics and animation. Traditional methods use markers or blendshapes to construct a mapping between the human and avatar faces. However, these approaches require a tedious 3D modeling process, and the performance relies on the modelers' experience. In this paper, we propose a brand-new solution to this cross-domain expression transfer problem via nonlinear expression embedding and expression domain translation. We first build low-dimensional latent spaces for the human and avatar facial expressions with variational autoencoder. Then we construct correspondences between the two latent spaces guided by geometric and perceptual constraints. Specifically, we design geometric correspondences to reflect geometric matching and utilize a triplet data structure to express users' perceptual preference of avatar expressions. A user-friendly method is proposed to automatically generate triplets for a system allowing users to easily and efficiently annotate the correspondences. Using both geometric and perceptual correspondences, we trained a network for expression domain translation from human to avatar. Extensive experimental results and user studies demonstrate that even nonprofessional users can apply our method to generate high-quality facial expression retargeting results with less time and effort.
University of Washington（华盛顿大学）；Microsoft Cloud and AI（微软云与人工智能部门）；Microsoft（微软）
Traditional methods for image-based 3D face reconstruction and facial motion retargeting fit a 3D morphable model (3DMM) to the face, which has limited modeling capacity and fail to generalize well to in-the-wild data. Use of deformation transfer or multilinear tensor as a personalized 3DMM for blendshape interpolation does not address the fact that facial expressions result in different local and global skin deformations in different persons. Moreover, existing methods learn a single albedo per user which is not enough to capture the expression-specific skin reflectance variations. We propose…
DynamicAvatars: Accurate Dynamic Facial Avatars Reconstruction and Precise Editing with Diffusion Models
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；National Engineering Research Center for Visual Information and Applications, Xi’an Jiaotong University（西安交通大学视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University（人机混合增强智能全国重点实验室）
DEGAS: Detailed Expressions on Full-Body Gaussian Avatars
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Prometheus Vision Technology Co., Ltd.（天视智能科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）；Swinburne University of Technology（斯威本科技大学）
High-fidelity facial avatar reconstruction from a monocular video is a significant research problem in computer graphics and computer vision. Recently, Neural Radiance Field (NeRF) has shown impressive novel view rendering results and has been considered for facial avatar reconstruction. However, the complex facial dynamics and missing 3D information in monocular videos raise significant challenges for faithful facial reconstruction. In this work, we propose a new method for NeRF-based facial avatar reconstruction that utilizes 3D-aware generative prior. Different from existing works that depe…
GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Tübingen（图宾根大学）；Meta Reality Labs（Meta现实实验室）；Technical University of Darmstadt（达姆施塔特工业大学）
Technical University of Munich（慕尼黑工业大学）；Facebook Reality Labs（Facebook现实实验室）
We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Digitally modeling and reconstructing a talking human is a key building-block for a variety of applications. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoints or head-poses is required. In contrast to state-of-the-art approaches that model the geometry and material properties explicitly, or are purely image-based, we introduce an implicit representation of the head based on scene representation networks. To handle the dyna…
Facebook Reality Labs（Facebook现实实验室）；University of Bonn（波恩大学）
Codec Avatars are a recent class of learned, photorealistic face models that accurately represent the geometry and texture of a person in 3D (i.e., for virtual reality), and are almost indistinguishable from video. In this paper we describe the first approach to animate these parametric models in real-time which could be deployed on commodity virtual reality hardware using audio and/or eye tracking. Our goal is to display expressive conversations between individuals that exhibit important social signals such as laughter and excitement solely from latent cues in our lossy input signals. To this…
Department of Control Science and Engineering, University of Shanghai for Science and Technology（控制科学与工程系，上海科学技术大学）；Business School, University of Shanghai for Science and Technology（商学院，上海科学技术大学）；School of Psychology and Neuroscience, University of Glasgow（心理学与神经科学学院，格拉斯哥大学）
Relightable 3D Avatar Reconstruction with Semantic-Adaptive Motion-Illumination Responses
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统全国重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；ZKTeco Co., Ltd.（熵基科技股份有限公司）
