Speech-driven 3D facial animation has garnered lots of attention thanks to its broad range of applications. Despite recent advancements in achieving realistic lip motion, current methods fail to capture the nuanced emotional undertones conveyed through speech and produce monotonous facial motion. These limitations result in blunt and repetitive facial animations, reducing user engagement and hindering their applicability. To address these challenges, we introduce DEEPTalk, a novel approach that generates diverse and emotionally rich 3D facial expressions directly from speech inputs. To achieve this, we first train DEE (Dynamic Emotion Embedding), which employs probabilistic contrastive learning to forge a joint emotion embedding space for both speech and facial motion. This probabilistic framework captures the uncertainty in interpreting emotions from speech and facial motion, enabling the derivation of emotion vectors from its multifaceted space. Moreover, to generate dynamic facial motion, we design TH-VQVAE (Temporally Hierarchical VQ-VAE) as an expressive and robust motion prior overcoming limitations of VAEs and VQ-VAEs. Utilizing these strong priors, we develop DEEPTalk, a talking head generator that non-autoregressively predicts codebook indices to create dynamic facial motion, incorporating a novel emotion consistency loss. Extensive experiments on various datasets demonstrate the effectiveness of our approach in creating diverse, emotionally expressive talking faces that maintain accurate lip-sync. Our project page is available at https://whwjdqls.github.io/deeptalk\_website/
EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models
University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology Beijing（北京科技大学）；The New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室模式识别新实验室）；The New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
As 3D facial avatars become more widely used for communication, it is critical that they faithfully convey emotion. Unfortunately, the best recent methods that regress parametric 3D face models from monocular images are unable to capture the full spectrum of facial expression, such as subtle or extreme emotions. We find the standard reconstruction metrics used for training (landmark reprojection error, photometric error, and face recognition loss) are insufficient to capture high-fidelity expressions. The result is facial geometries that do not match the emotional content of the input image. W…
LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representation
University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Academy（中关村学院）；National University of Singapore（新加坡国立大学）；Chongqing University（重庆大学）；University of Science and Technology of China（中国科学技术大学）
MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation
Zhejiang Lab（之江实验室）；Li Auto（理想汽车）；Harbin Institute of Technology（哈尔滨工业大学）；Zhejiang University（浙江大学）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）
Dramatic progress has been made in animating individual characters. However, we still lack automatic control over activities between characters, especially those involving interactions. In this paper, we present a novel energy-based framework to sample and synthesize animations by associating the characters' body motions, facial expressions, and social relations. We propose a Spatial-Temporal And-Or graph (ST-AOG), a stochastic grammar model, to encode the contextual relationship between motion, emotion, and relation, forming a triangle in a conditional random field. We train our model from a …
Controllable Expressive 3D Facial Animation via Diffusion in a Unified Multimodal Space
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Laboratory for Big Data and Decision, School of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院大数据与决策实验室）
École Polytechnique Montréal（蒙特利尔综合理工学院）；Université de Montréal（蒙特利尔大学）
We present techniques for improving performance driven facial animation, emotion recognition, and facial key-point or landmark prediction using learned identity invariant representations. Established approaches to these problems can work well if sufficient examples and labels for a particular identity are available and factors of variation are highly controlled. However, labeled examples of facial expressions, emotions and key-points for new individuals are difficult and costly to obtain. In this paper we improve the ability of techniques to generalize to new and unseen individuals by explicit…
PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Psyche AI.INC（Psyche AI公司）；HKUST（香港科技大学）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院计算智能研究所）；SCSE, FIE, M.U.S.T（M.U.S.T的SCSE、FIE部门）
We present a deep learning framework for real-time speech-driven 3D facial animation from just raw waveforms. Our deep neural network directly maps an input sequence of speech audio to a series of micro facial action unit activations and head rotations to drive a 3D blendshape face model. In particular, our deep model is able to learn the latent representations of time-varying contextual information and affective states within the speech. Hence, our model not only activates appropriate facial action units at inference to depict different utterance generating actions, in the form of lip movemen…
EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
Renmin University of China（中国人民大学）；Nanjing University of Science and Technology（南京理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Psyche AI Inc.（赛琪人工智能公司）；Chinese Academy of Sciences（中国科学院）；Tsinghua University（清华大学）
Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huadian (Beijing) Co-Generation Co., Ltd.（华电（北京）热电有限公司）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Impressive progress has been made in audio-driven 3D facial animation recently, but synthesizing 3D talking-head with rich emotion is still unsolved. This is due to the lack of 3D generative models and available 3D emotional dataset with synchronized audios. To address this, we introduce 3D-TalkEmo, a deep neural network that generates 3D talking head animation with various emotions. We also create a large 3D dataset with synchronized audios and videos, rich corpus, as well as various emotion states of different persons with the sophisticated 3D face reconstruction methods. In the emotion gene…
FreeTalk: Emotional Topology-Free 3D Talking Heads
University of Copenhagen（哥本哈根大学）；IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems（IMT北欧、 Mines-Telecom学院、数字系统中心）；Univ. Lille, CNRS, Centrale Lille, Institut Mines-Telecom, UMR 9189 CRIStAL, F-59000 Lille, France（里尔大学、CNRS、里尔中央大学、Mines-Telecom学院、UMR 9189 CRIStAL、法国里尔）
Animating portraits using speech has received growing attention in recent years, with various creative and practical use cases. An ideal generated video should have good lip sync with the audio, natural facial expressions and head motions, and high frame quality. In this work, we present SPACE, which uses speech and a single image to generate high-resolution, and expressive videos with realistic head pose, without requiring a driving video. It uses a multi-stage approach, combining the controllability of facial landmarks with the high-quality synthesis power of a pretrained face generator. SPA…
This paper presents FaceXHuBERT, a text-less speech-driven 3D facial animation generation method that allows to capture personalized and subtle cues in speech (e.g. identity, emotion and hesitation). It is also very robust to background noise and can handle audio recorded in a variety of situations (e.g. multiple people speaking). Recent approaches employ end-to-end deep learning taking into account both audio and text as input to generate facial animation for the whole face. However, scarcity of publicly available expressive audio-3D facial animation datasets poses a major bottleneck. The res…
Netease Fuxi AI Lab（网易伏羲人工智能实验室）；University of Technology Sydney（悉尼科技大学）
In this paper, we propose a novel text-based talking-head video generation framework that synthesizes high-fidelity facial expressions and head motions in accordance with contextual sentiments as well as speech rhythm and pauses. To be specific, our framework consists of a speaker-independent stage and a speaker-specific stage. In the speaker-independent stage, we design three parallel networks to generate animation parameters of the mouth, upper face, and head from texts, separately. In the speaker-specific stage, we present a 3D face model guided attention network to synthesize videos tailor…
AU-Blendshape for Fine-grained Stylized 3D Facial Expression Manipulation
Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）；North China University of Technology（华北理工大学）；University of Washington（华盛顿大学）；University of Copenhagen（哥本哈根大学）
EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Xi’an Key Laboratory of Big Data and Intelligent Vision, Xidian University（西安电子科技大学西安大数据与智能视觉重点实验室）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）
MagicAnime: A Hierarchically Annotated, Multimodal and Multitasking Dataset with Benchmarks for Cartoon Animation Generation
National Centre for Computer Animation, Bournemouth University（伯恩茅斯大学计算机动画国家中心）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Department of Computer Science and Information Engineering, National Cheng Kung University（国立成功大学计算机科学与信息工程系）
SPEAK: Speech-Driven Pose and Emotion-Adjustable Talking Head Generation
Ping An Technology（平安科技）；Southeast University（东南大学）；Sun Yat-sen University（中山大学）；Nanjing University of Science and Technology（南京理工大学）；East China University of Science and Technology（华东理工大学）；University of Shanghai for Science and Technology（上海理工大学）
