音频驱动肖像动画旨在合成由给定音频条件控制的肖像视频。生成高保真、多模态的肖像视频具有广泛的应用场景。现有方法尝试通过训练不同模型或从给定视频中采样信号，来捕捉不同运动模式并生成高保真肖像视频。然而，这类方法缺乏对口型同步与其他动作（如头部姿态、眨眼）之间的相关性学习，通常会导致生成结果不自然。本文提出了一个统一系统，可生成多人物、多样化、高保真的说话人肖像。该方法包含三个阶段：1）具备双注意力的单次映射网络（Mapping-Once network with Dual Attentions，MODA）从给定音频生成说话表征，其中设计的双注意力模块用于编码精准的嘴部动作与多样化模态；2）面部合成网络生成密集且细节丰富的人脸关键点；3）时序引导渲染器合成稳定的视频。大量评估实验表明，与现有方法相比，该系统生成的视频肖像更自然、更真实。
英文摘要：
Audio-driven portrait animation aims to synthesize portrait videos that are conditioned by given audio. Animating high-fidelity and multimodal video portraits has a variety of applications. Previous methods have attempted to capture different motion modes and generate high-fidelity portrait videos by training different models or sampling signals from given videos. However, lacking correlation learning between lip-sync and other movements (e.g., head pose/eye blinking) usually leads to unnatural results. In this paper, we propose a unified system for multi-person, diverse, and high-fidelity talking portrait generation. Our method contains three stages, i.e., 1) Mapping-Once network with Dual Attentions (MODA) generates talking representation from given audio. In MODA, we design a dual-attention module to encode accurate mouth movements and diverse modalities. 2) Facial composer network generates dense and detailed face landmarks, and 3) temporal-guided renderer syntheses stable videos. Extensive evaluations demonstrate that the proposed system produces more natural and realistic video portraits compared to previous methods.
Animating portraits using speech has received growing attention in recent years, with various creative and practical use cases. An ideal generated video should have good lip sync with the audio, natural facial expressions and head motions, and high frame quality. In this work, we present SPACE, which uses speech and a single image to generate high-resolution, and expressive videos with realistic head pose, without requiring a driving video. It uses a multi-stage approach, combining the controllability of facial landmarks with the high-quality synthesis power of a pretrained face generator. SPA…
National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；North China University of Technology（北方工业大学）
Image matting plays an important role in image and video editing. However, the formulation of image matting is inherently ill-posed. Traditional methods usually employ interaction to deal with the image matting problem with trimaps and strokes, and cannot run on the mobile phone in real-time. In this paper, we propose a real-time automatic deep matting approach for mobile devices. By leveraging the densely connected blocks and the dilated convolution, a light full convolutional network is designed to predict a coarse binary mask for portrait images. And a feathering block, which is edge-preser…
SynergyWarpNet: Attention-Guided Cooperative Warping for Neural Portrait Animation
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能联合实验室，人工智能研究院，上海交通大学）；Defense Innovation Institute, Academy of Military Science（国防创新研究院，军事科学院）；Intelligent Game and Decision Laboratory（智能游戏与决策实验室）
GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；NIO（蔚来）；Banma（斑马）；Beihang University（北京航空航天大学）
Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation
Fujian Provincial Natural Science Foundation of China（福建省自然科学基金委员会）；Giant Interactive Group Inc.（巨匠互动集团有限公司）；National Natural Science Foundation of China（国家自然科学基金委员会）
Millions of images of human faces are captured every single day; but these photographs portray the likeness of an individual with a fixed pose, expression, and appearance. Portrait image animation enables the post-capture adjustment of these attributes from a single image while maintaining a photorealistic reconstruction of the subject's likeness or identity. Still, current methods for portrait image animation are typically based on 2D warping operations or manipulations of a 2D generative adversarial network (GAN) and lack explicit mechanisms to enforce multi-view consistency. Thus these meth…
University of Trento（特伦托大学）；Zhejiang University（浙江大学）；Shandong University（山东大学）；ETH Zurich（苏黎世联邦理工学院）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；Illinois Institute of Technology（伊利诺伊理工学院）
This paper proposes a gaze correction and animation method for high-resolution, unconstrained portrait images, which can be trained without the gaze angle and the head pose annotations. Common gaze-correction methods usually require annotating training data with precise gaze, and head pose information. Solving this problem using an unsupervised method remains an open problem, especially for high-resolution face images in the wild, which are not easy to annotate with gaze and head pose labels. To address this issue, we first create two new portrait datasets: CelebGaze and high-resolution CelebH…
This paper presents a neural rendering method for controllable portrait video synthesis. Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), has enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of a scene with such scene representations is still challenging. In this work, we design a system that enables both novel view synthesis for portrait video, including the human subject and the scene background, and explicit control of the facial expressions through a low…
To the best of our knowledge, we first present a live system that generates personalized photorealistic talking-head animation only driven by audio signals at over 30 fps. Our system contains three stages. The first stage is a deep neural network that extracts deep audio features along with a manifold projection to project the features to the target person's speech space. In the second stage, we learn facial dynamics and motions from the projected audio features. The predicted motions include head poses and upper body motions, where the former is generated by an autoregressive probabilistic mo…
MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation
Zhejiang Lab（之江实验室）；Li Auto（理想汽车）；Harbin Institute of Technology（哈尔滨工业大学）；Zhejiang University（浙江大学）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）
