Recent works in implicit representations, such as Neural Radiance Fields (NeRF), have advanced the generation of realistic and animatable head avatars from video sequences. These implicit methods are still confronted by visual artifacts and jitters, since the lack of explicit geometric constraints poses a fundamental challenge in accurately modeling complex facial deformations. In this paper, we introduce Dynamic Tetrahedra (DynTet), a novel hybrid representation that encodes explicit dynamic meshes by neural networks to ensure geometric consistency across various motions and viewpoints. DynTet is parameterized by the coordinate-based networks which learn signed distance, deformation, and material texture, anchoring the training data into a predefined tetrahedra grid. Leveraging Marching Tetrahedra, DynTet efficiently decodes textured meshes with a consistent topology, enabling fast rendering through a differentiable rasterizer and supervision via a pixel loss. To enhance training efficiency, we incorporate classical 3D Morphable Models to facilitate geometry learning and define a canonical space for simplifying texture learning. These advantages are readily achievable owing to the effective geometric representation employed in DynTet. Compared with prior works, DynTet demonstrates significant improvements in fidelity, lip synchronization, and real-time performance according to various metrics. Beyond producing stable and visually appealing synthesis videos, our method also outputs the dynamic meshes which is promising to enable many emerging applications.
TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting
Beihang University（北京航空航天大学）；Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）；Griffith University（格里菲斯大学）；RIKEN AIP（理化学研究所人工智能项目）；The University of Tokyo（东京大学）
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；PhiGent Robotics（鉴智机器人）
Talking head synthesis is an emerging technology with wide applications in film dubbing, virtual avatars and online education. Recent NeRF-based methods generate more natural talking videos, as they better capture the 3D structural information of faces. However, a specific model needs to be trained for each identity with a large dataset. In this paper, we propose Dynamic Facial Radiance Fields (DFRF) for few-shot talking head synthesis, which can rapidly generalize to an unseen identity with few training data. Different from the existing NeRF-based methods which directly encode the 3D geometry…
GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院，教育部高维信息智能感知与系统重点实验室，PCA实验室）；Tsientang Institute for Advanced Study（浙江大学前沿交叉研究院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
University of Science and Technology of China（中国科学技术大学）；Beijing Dilusense Technology Corporation（北京谛听视界科技有限公司）；Zhejiang University（浙江大学）；Tsinghua University（清华大学）
Generating high-fidelity talking head video by fitting with the input audio sequence is a challenging problem that receives considerable attentions recently. In this paper, we address this problem with the aid of neural scene representation networks. Our method is completely different from existing methods that rely on intermediate representations like 2D landmarks or 3D face models to bridge the gap between audio input and video output. Specifically, the feature of input audio signal is directly fed into a conditional implicit function to generate a dynamic neural radiance field, from which a…
Toward Fine-Grained Facial Control in 3D Talking Head Generation
School of Cyber Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李科金医学院）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感与信息工程学院）；Purple Mountain Laboratories, Nanjing（南京紫金山实验室）；Engineering Research Center of Blockchain Application, Supervision And Management (Southeast University), Ministry of Education（教育部区块链应用、监督与管理工程研究中心（东南大学））；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
EmbedTalk: Talking Head Synthesis using Gaussian Embeddings
School of Computer Science, University of Leeds（利兹大学计算机科学学院）；Leeds Institute of Medical Education, School of Medicine, University of Leeds（利兹大学医学教育研究所）
SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting
School of Information, Renmin University of China（中国人民大学信息学院）；School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与通信工程学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；School of Economics and Management, Tsinghua University（清华大学经济管理学院）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
SyncAnimation: A Real-Time End-to-End Framework for Audio-Driven Human Pose and Talking Head Animation
AiShiWeiLai AI Research（AiShiWeiLai AI研究院）；South China University of Technology（华南理工大学）；Beijing Institute of Technology（北京理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
S^3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Alibaba Group（阿里巴巴集团）
EmoGene: Audio-Driven Emotional 3D Talking-Head Generation
Khoury College of Computer Science, Northeastern University, USA（东北大学Khoury计算机科学学院）；Department of Electrical and Computer Engineering, Northeastern University, USA（东北大学电气与计算机工程系）
PASE: Phoneme-Aware Speech Encoder to Improve Lip Sync Accuracy for Talking Head Synthesis
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University（航空航天信息安全部分和可信计算重点实验室、教育部、网络安全科学与工程学院、武汉大学）；School of Cyber Science and Engineering, Wuhan University（网络安全科学与工程学院、武汉大学）；School of Police Information, Shandong Police College（警务信息学院、山东警察学院）
Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions
Engineering Sciences, Homi Bhabha National Institute Training School Complex（工程科学，霍米·巴赫瓦全国研究所培训学校）；Computer and Informatics Group, VECC 1/AF（计算机与信息组，VECC 1/AF）
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Alibaba Group（阿里巴巴集团）；Nanjing University（南京大学）
Audio-Plane: Audio Factorization Plane Gaussian Splatting for Real-Time Talking Head Synthesis
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；John A. Paulson School of Engineering and Applied Sciences at Harvard University（哈佛大学约翰·A·保罗森工程与应用科学学院）；PhiGent Robotics（PhiGent机器人技术公司）；Beijing National Research Center for Information Science and Technology (BNRist) and the Department of Automation, Tsinghua University（北京信息科学与技术国家研究中心（BNRist）和清华大学自动化系）
FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motions
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
AIOZ；University of Liverpool（利物浦大学）；VinAI Research（VinAI研究院）
Audio-driven talking head animation is a challenging research topic with many real-world applications. Recent works have focused on creating photo-realistic 2D animation, while learning different talking or singing styles remains an open problem. In this paper, we present a new method to generate talking head animation with learnable style references. Given a set of style reference frames, our framework can reconstruct 2D talking head animation based on a single input image and an audio stream. Our method first produces facial landmarks motion from the audio stream and constructs the intermedi…
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2 School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China（1 中国科学院信息工程研究所，北京，中国 2 中国科学院大学网络空间安全学院，北京，中国）
One-shot talking head generation produces lip-sync talking heads based on arbitrary audio and one source face. To guarantee the naturalness and realness, recent methods propose to achieve free pose control instead of simply editing mouth areas. However, existing methods do not preserve accurate identity of source face when generating head motions. To solve the identity mismatch problem and achieve high-quality free pose control, we present One-shot Pose-controllable Talking head generation network (OPT). Specifically, the Audio Feature Disentanglement Module separates content features from aud…
Renmin University of China（中国人民大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Psyche AI Inc.（Psyche AI公司）；Chinese Academy of Sciences（中国科学院）；Tsinghua University（清华大学）
Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. Traditional Generative Adversarial Networks (GAN) struggle to maintain consistent facial identity, while Neural Radiance Fields (NeRF) methods, although they can address this issue, often produce mismatched lip movements, inadequate facial expressions, and unstable head poses. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a fundamental flaw, …
TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles
Tsinghua University（清华大学）；BNRist（北京信息科学与技术国家研究中心）；State Key Laboratory of Intelligent Technology and Systems（智能技术与系统国家重点实验室）；THUAI（清华大学人工智能国际研究院）；Fuxi AI Lab（伏羲人工智能实验室）；Netease（网易公司）；School of Computer Science, the University of Queensland（昆士兰大学计算机科学学院）
IIIT-Hyderabad（印度海得拉巴国际信息技术学院）；University of Bath（巴斯大学）
We leverage the modern advancements in talking head generation to propose an end-to-end system for talking head video compression. Our algorithm transmits pivot frames intermittently while the rest of the talking head video is generated by animating them. We use a state-of-the-art face reenactment network to detect key points in the non-pivot frames and transmit them to the receiver. A dense flow is then calculated to warp a pivot frame to reconstruct the non-pivot ones. Transmitting key points instead of full frames leads to significant compression. We propose a novel algorithm to adaptively …
We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables …
While recent advances in deep neural networks have made it possible to render high-quality images, generating photo-realistic and personalized talking head remains challenging. With given audio, the key to tackling this task is synchronizing lip movement and simultaneously generating personalized attributes like head movement and eye blink. In this work, we observe that the input audio is highly correlated to lip motion while less correlated to other personalized attributes (e.g., head movements). Inspired by this, we propose a novel framework based on neural radiance field to pursue high-fide…
Imperial College London（帝国理工学院）；Huawei Technologies（华为技术有限公司）；University of Sussex（萨塞克斯大学）
We present Free-HeadGAN, a person-generic neural talking head synthesis system. We show that modeling faces with sparse 3D facial landmarks are sufficient for achieving state-of-the-art generative performance, without relying on strong statistical priors of the face, such as 3D Morphable Models. Apart from 3D pose and facial expressions, our method is capable of fully transferring the eye gaze, from a driving actor to a source identity. Our complete pipeline consists of three components: a canonical 3D key-point estimator that regresses 3D pose and expression-related deformations, a gaze estim…
We present a novel one-shot talking head synthesis method that achieves disentangled and fine-grained control over lip motion, eye gaze&blink, head pose, and emotional expression. We represent different motions via disentangled latent representations and leverage an image generator to synthesize talking heads from them. To effectively disentangle each motion factor, we propose a progressive disentangled representation learning strategy by separating the factors in a coarse-to-fine manner, where we first extract unified motion feature from the driving signal, and then isolate each fine-grained …
Beijing University of Posts and Telecommunications（北京邮电大学）；DeepScience Tech Ltd.（深度科学科技有限公司）
Audio driven talking head synthesis is a challenging task that attracts increasing attention in recent years. Although existing methods based on 2D landmarks or 3D face models can synthesize accurate lip synchronization and rhythmic head pose for arbitrary identity, they still have limitations, such as the cut feeling in the mouth mapping and the lack of skin highlights. The morphed region is blurry compared to the surrounding face. A Keypoint Based Enhancement (KPBE) method is proposed for audio driven free view talking head synthesis to improve the naturalness of the generated video. Firstly…
We propose a novel approach for few-shot talking-head synthesis. While recent works in neural talking heads have produced promising results, they can still produce images that do not preserve the identity of the subject in source images. We posit this is a result of the entangled representation of each subject in a single latent code that models 3D shape information, identity cues, colors, lighting and even background details. In contrast, we propose to factorize the representation of a subject into its spatial and style components. Our method generates a target frame in two steps. First, it p…
