Audio-driven emotional 3D face animation aims to generate emotionally expressive talking heads with synchronized lip movements. However, previous research has often overlooked the influence of diverse emotions on facial expressions or proved unsuitable for driving MetaHuman models. In response to this deficiency, we introduce EmoFace, a novel audio-driven methodology for creating facial animations with vivid emotional dynamics. Our approach can generate facial expressions with multiple emotions, and has the ability to generate random yet natural blinks and eye movements, while maintaining accurate lip synchronization. We propose independent speech encoders and emotion encoders to learn the relationship between audio, emotion and corresponding facial controller rigs, and finally map into the sequence of controller values. Additionally, we introduce two post-processing techniques dedicated to enhancing the authenticity of the animation, particularly in blinks and eye movements. Furthermore, recognizing the scarcity of emotional audio-visual data suitable for MetaHuman model manipulation, we contribute an emotional audio-visual dataset and derive control parameters for each frames. Our proposed methodology can be applied in producing dialogues animations of non-playable characters (NPCs) in video games, and driving avatars in virtual reality environments. Our further quantitative and qualitative experiments, as well as an user study comparing with existing researches show that our approach demonstrates superior results in driving 3D facial models. The code and sample data are available at https://github.com/SJTU-Lucy/EmoFace.
LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representation
University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Academy（中关村学院）；National University of Singapore（新加坡国立大学）；Chongqing University（重庆大学）；University of Science and Technology of China（中国科学技术大学）
EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models
University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology Beijing（北京科技大学）；The New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室模式识别新实验室）；The New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Impressive progress has been made in audio-driven 3D facial animation recently, but synthesizing 3D talking-head with rich emotion is still unsolved. This is due to the lack of 3D generative models and available 3D emotional dataset with synchronized audios. To address this, we introduce 3D-TalkEmo, a deep neural network that generates 3D talking head animation with various emotions. We also create a large 3D dataset with synchronized audios and videos, rich corpus, as well as various emotion states of different persons with the sophisticated 3D face reconstruction methods. In the emotion gene…
Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huadian (Beijing) Co-Generation Co., Ltd.（华电（北京）热电有限公司）
EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
Renmin University of China（中国人民大学）；Nanjing University of Science and Technology（南京理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Psyche AI Inc.（赛琪人工智能公司）；Chinese Academy of Sciences（中国科学院）；Tsinghua University（清华大学）
We present a deep learning framework for real-time speech-driven 3D facial animation from just raw waveforms. Our deep neural network directly maps an input sequence of speech audio to a series of micro facial action unit activations and head rotations to drive a 3D blendshape face model. In particular, our deep model is able to learn the latent representations of time-varying contextual information and affective states within the speech. Hence, our model not only activates appropriate facial action units at inference to depict different utterance generating actions, in the form of lip movemen…
MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation
Zhejiang Lab（之江实验室）；Li Auto（理想汽车）；Harbin Institute of Technology（哈尔滨工业大学）；Zhejiang University（浙江大学）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）
PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Psyche AI.INC（Psyche AI公司）；HKUST（香港科技大学）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院计算智能研究所）；SCSE, FIE, M.U.S.T（M.U.S.T的SCSE、FIE部门）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
As 3D facial avatars become more widely used for communication, it is critical that they faithfully convey emotion. Unfortunately, the best recent methods that regress parametric 3D face models from monocular images are unable to capture the full spectrum of facial expression, such as subtle or extreme emotions. We find the standard reconstruction metrics used for training (landmark reprojection error, photometric error, and face recognition loss) are insufficient to capture high-fidelity expressions. The result is facial geometries that do not match the emotional content of the input image. W…
Controllable Expressive 3D Facial Animation via Diffusion in a Unified Multimodal Space
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Laboratory for Big Data and Decision, School of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院大数据与决策实验室）
Netease Fuxi AI Lab（网易伏羲人工智能实验室）；University of Technology Sydney（悉尼科技大学）
In this paper, we propose a novel text-based talking-head video generation framework that synthesizes high-fidelity facial expressions and head motions in accordance with contextual sentiments as well as speech rhythm and pauses. To be specific, our framework consists of a speaker-independent stage and a speaker-specific stage. In the speaker-independent stage, we design three parallel networks to generate animation parameters of the mouth, upper face, and head from texts, separately. In the speaker-specific stage, we present a 3D face model guided attention network to synthesize videos tailor…
FreeTalk: Emotional Topology-Free 3D Talking Heads
University of Copenhagen（哥本哈根大学）；IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems（IMT北欧、 Mines-Telecom学院、数字系统中心）；Univ. Lille, CNRS, Centrale Lille, Institut Mines-Telecom, UMR 9189 CRIStAL, F-59000 Lille, France（里尔大学、CNRS、里尔中央大学、Mines-Telecom学院、UMR 9189 CRIStAL、法国里尔）
GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院，教育部高维信息智能感知与系统重点实验室，PCA实验室）；Tsientang Institute for Advanced Study（浙江大学前沿交叉研究院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
AU-Blendshape for Fine-grained Stylized 3D Facial Expression Manipulation
Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）；North China University of Technology（华北理工大学）；University of Washington（华盛顿大学）；University of Copenhagen（哥本哈根大学）
EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Xi’an Key Laboratory of Big Data and Intelligent Vision, Xidian University（西安电子科技大学西安大数据与智能视觉重点实验室）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）
SPEAK: Speech-Driven Pose and Emotion-Adjustable Talking Head Generation
Ping An Technology（平安科技）；Southeast University（东南大学）；Sun Yat-sen University（中山大学）；Nanjing University of Science and Technology（南京理工大学）；East China University of Science and Technology（华东理工大学）；University of Shanghai for Science and Technology（上海理工大学）
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Meituan New Laboratory of Pattern Recognition (NLPR), CASIA（美团模式识别新实验室（NLPR），中国科学院自动化所）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Meituan Division of AMC and Department of ECE, HKUST（美团AMC部门和香港科技大学电子与计算机工程系）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）；Meituan School of Artificial Intelligence, University of Chinese Academy of Sciences（美团人工智能学院，中国科学院大学）；Meituan（美团）
See the Emotion: A Facial Emoji Proxy Modeling for EEG Emotion Recognition
Hefei University of Technology（合肥工业大学）；PLA Information Engineering University（中国人民解放军信息工程大学）；University of Science and Technology of China（中国科学技术大学）；MBZUAI
