近年来，情感说话人脸生成受到了广泛关注。然而，现有方法仅采用独热编码、图像或音频作为情感条件，因而在实际应用中缺乏灵活控制能力，并且由于语义有限，难以处理未见过的情感风格。这些方法要么忽略了一次性（one-shot）设置，要么忽视了生成人脸的质量。在本文中，我们提出了一个更灵活且更具泛化能力的框架。具体而言，我们在文本提示中补充情感风格，并使用一个对齐的多模态情感编码器（Aligned Multi-modal Emotion encoder）将文本、图像和音频三种情感模态嵌入到一个统一空间中，该空间继承了来自CLIP的丰富语义先验。因此，有效的多模态情感空间学习使我们的方法能够在测试阶段支持任意情感模态，并可泛化到未见过的情感风格。此外，我们提出了一个情感感知的音频到3DMM转换器（Emotion-aware Audio-to-3DMM Convertor），用于将情感条件与音频序列连接到结构表示。随后设计了一个基于风格的保真情感人脸生成器（High-fidelity Emotional Face generator），用于生成任意高分辨率的逼真身份。我们的纹理生成器以残差方式分层学习流场和动态人脸。大量实验证明了我们的方法在情感控制方面的灵活性与泛化能力，以及高质量人脸合成的有效性。
英文摘要：
Recently, emotional talking face generation has received considerable attention. However, existing methods only adopt one-hot coding, image, or audio as emotion conditions, thus lacking flexible control in practical applications and failing to handle unseen emotion styles due to limited semantics. They either ignore the one-shot setting or the quality of generated faces. In this paper, we propose a more flexible and generalized framework. Specifically, we supplement the emotion style in text prompts and use an Aligned Multi-modal Emotion encoder to embed the text, image, and audio emotion modality into a unified space, which inherits rich semantic prior from CLIP. Consequently, effective multi-modal emotion space learning helps our method support arbitrary emotion modality during testing and could generalize to unseen emotion styles. Besides, an Emotion-aware Audio-to-3DMM Convertor is proposed to connect the emotion condition and the audio sequence to structural representation. A followed style-based High-fidelity Emotional Face generator is designed to generate arbitrary high-resolution realistic identities. Our texture generator hierarchically learns flow fields and animated faces in a residual manner. Extensive experiments demonstrate the flexibility and generalization of our method in emotion control and the effectiveness of high-quality face synthesis.
Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）；University of Sydney（悉尼大学）；Monash University（莫纳什大学）
Although significant progress has been made to audio-driven talking face generation, existing methods either neglect facial emotion or cannot be applied to arbitrary subjects. In this paper, we propose the Emotion-Aware Motion Model (EAMM) to generate one-shot emotional talking faces by involving an emotion source video. Specifically, we first propose an Audio2Facial-Dynamics module, which renders talking faces from audio-driven unsupervised zero- and first-order key-points motion. Then through exploring the motion model's properties, we further propose an Implicit Emotion Displacement Learner…
GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院，教育部高维信息智能感知与系统重点实验室，PCA实验室）；Tsientang Institute for Advanced Study（浙江大学前沿交叉研究院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
Carnegie Mellon University（卡内基梅隆大学）；National Institute of Informatics（国立信息学研究所）；A*STAR（新加坡科技研究局）
Several works have developed end-to-end pipelines for generating lip-synced talking faces with various real-world applications, such as teaching and language translation in videos. However, these prior works fail to create realistic-looking videos since they focus little on people's expressions and emotions. Moreover, these methods' effectiveness largely depends on the faces in the training dataset, which means they may not perform well on unseen faces. To mitigate this, we build a talking face generation framework conditioned on a categorical emotion to generate videos with appropriate expres…
Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation
School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Ningbo Global Innovation Center, Zhejiang University（浙江大学宁波全球创新中心）
SPEAK: Speech-Driven Pose and Emotion-Adjustable Talking Head Generation
Ping An Technology（平安科技）；Southeast University（东南大学）；Sun Yat-sen University（中山大学）；Nanjing University of Science and Technology（南京理工大学）；East China University of Science and Technology（华东理工大学）；University of Shanghai for Science and Technology（上海理工大学）
We present a novel one-shot talking head synthesis method that achieves disentangled and fine-grained control over lip motion, eye gaze&blink, head pose, and emotional expression. We represent different motions via disentangled latent representations and leverage an image generator to synthesize talking heads from them. To effectively disentangle each motion factor, we propose a progressive disentangled representation learning strategy by separating the factors in a coarse-to-fine manner, where we first extract unified motion feature from the driving signal, and then isolate each fine-grained …
AUHead: Realistic Emotional Talking Head Generation via Action Units Control
University of the Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；State Key Laboratory of Communication Content Cognition, People’s Daily Online（人民日報網通信內容認知重點實驗室）
EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Xi’an Key Laboratory of Big Data and Intelligent Vision, Xidian University（西安电子科技大学西安大数据与智能视觉重点实验室）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）
Netease Fuxi AI Lab（网易伏羲人工智能实验室）；University of Technology Sydney（悉尼科技大学）
In this paper, we propose a novel text-based talking-head video generation framework that synthesizes high-fidelity facial expressions and head motions in accordance with contextual sentiments as well as speech rhythm and pauses. To be specific, our framework consists of a speaker-independent stage and a speaker-specific stage. In the speaker-independent stage, we design three parallel networks to generate animation parameters of the mouth, upper face, and head from texts, separately. In the speaker-specific stage, we present a 3D face model guided attention network to synthesize videos tailor…
Affective Image Editing: Shaping Emotional Factors via Text Descriptions
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China（北京邮电大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；State Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, China（多媒体信息处理国家重点实验室和视觉技术国家工程研究中心，北京大学计算机学院）
EmoGene: Audio-Driven Emotional 3D Talking-Head Generation
Khoury College of Computer Science, Northeastern University, USA（东北大学Khoury计算机科学学院）；Department of Electrical and Computer Engineering, Northeastern University, USA（东北大学电气与计算机工程系）
Second University of Naples（那不勒斯第二大学）；IIASS；Brno University of Technology（布尔诺理工大学）；EUP Mataró（马塔罗EUP）
This work defines a procedure for collecting naturally induced emotional facial expressions through the vision of movie excerpts with high emotional contents and reports experimental data ascertaining the effects of emotions on memory word recognition tasks. The induced emotional states include the four basic emotions of sadness, disgust, happiness, and surprise, as well as the neutral emotional state. The resulting database contains both thermal and visible emotional facial expressions, portrayed by forty Italian subjects and simultaneously acquired by appropriately synchronizing a thermal an…
FreeTalk: Emotional Topology-Free 3D Talking Heads
University of Copenhagen（哥本哈根大学）；IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems（IMT北欧、 Mines-Telecom学院、数字系统中心）；Univ. Lille, CNRS, Centrale Lille, Institut Mines-Telecom, UMR 9189 CRIStAL, F-59000 Lille, France（里尔大学、CNRS、里尔中央大学、Mines-Telecom学院、UMR 9189 CRIStAL、法国里尔）
Visual emotion expression plays an important role in audiovisual speech communication. In this work, we propose a novel approach to rendering visual emotion expression in speech-driven talking face generation. Specifically, we design an end-to-end talking face generation system that takes a speech utterance, a single face image, and a categorical emotion label as input to render a talking face video synchronized with the speech and expressing the conditioned emotion. Objective evaluation on image quality, audiovisual synchronization, and visual emotion expression shows that the proposed system…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Impressive progress has been made in audio-driven 3D facial animation recently, but synthesizing 3D talking-head with rich emotion is still unsolved. This is due to the lack of 3D generative models and available 3D emotional dataset with synchronized audios. To address this, we introduce 3D-TalkEmo, a deep neural network that generates 3D talking head animation with various emotions. We also create a large 3D dataset with synchronized audios and videos, rich corpus, as well as various emotion states of different persons with the sophisticated 3D face reconstruction methods. In the emotion gene…
Despite the significant progress in recent years, very few of the AI-based talking face generation methods attempt to render natural emotions. Moreover, the scope of the methods is majorly limited to the characteristics of the training dataset, hence they fail to generalize to arbitrary unseen faces. In this paper, we propose a one-shot facial geometry-aware emotional talking face generation method that can generalize to arbitrary faces. We propose a graph convolutional neural network that uses speech content feature, along with an independent emotion input to generate emotion and speech-induc…
PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Psyche AI.INC（Psyche AI公司）；HKUST（香港科技大学）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院计算智能研究所）；SCSE, FIE, M.U.S.T（M.U.S.T的SCSE、FIE部门）
Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huadian (Beijing) Co-Generation Co., Ltd.（华电（北京）热电有限公司）
READ-Net: Clarifying Emotional Ambiguity via Adaptive Feature Recalibration for Audio-Visual Depression Detection
College of Computer Science and Technology, China University of Petroleum (East China)（中国石油大学（华东）计算机科学与技术学院）；College of Computer Science, Nankai University（南开大学计算机科学学院）；School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）
