Toward Fine-Grained Facial Control in 3D Talking Head Generation
School of Cyber Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李科金医学院）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感与信息工程学院）；Purple Mountain Laboratories, Nanjing（南京紫金山实验室）；Engineering Research Center of Blockchain Application, Supervision And Management (Southeast University), Ministry of Education（教育部区块链应用、监督与管理工程研究中心（东南大学））；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
Shaoyang Xie, Xiaofeng Cong, Baosheng Yu, Zhipeng Gui, Jie Gui, Yuan Yan Tang, James Tin-Yau Kwok
Audio-driven talking head generation is a core component of digital avatars, and 3D Gaussian Splatting has shown strong performance in real-time rendering of high-fidelity talking heads. However, achieving precise control over fine-grained facial movements remains a significant challenge, particularly due to lip-synchronization inaccuracies and facial jitter, both of which can contribute to the uncanny valley effect. To address these challenges, we propose Fine-Grained 3D Gaussian Splatting (FG-3DGS), a novel framework that enables temporally consistent and high-fidelity talking head generation. Our method introduces a frequency-aware disentanglement strategy to explicitly model facial regions based on their motion characteristics. Low-frequency regions, such as the cheeks, nose, and forehead, are jointly modeled using a standard MLP, while high-frequency regions, including the eyes and mouth, are captured separately using a dedicated network guided by facial area masks. The predicted motion dynamics, represented as Gaussian deltas, are applied to the static Gaussians to generate the final head frames, which are rendered via a rasterizer using frame-specific camera parameters. Additionally, a high-frequency-refined post-rendering alignment mechanism, learned from large-scale audio-video pairs by a pretrained model, is incorporated to enhance per-frame generation and achieve more accurate lip synchronization. Extensive experiments on widely used datasets for talking head generation demonstrate that our method outperforms recent state-of-the-art approaches in producing high-fidelity, lip-synced talking head videos.
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；PhiGent Robotics（鉴智机器人）
Talking head synthesis is an emerging technology with wide applications in film dubbing, virtual avatars and online education. Recent NeRF-based methods generate more natural talking videos, as they better capture the 3D structural information of faces. However, a specific model needs to be trained for each identity with a large dataset. In this paper, we propose Dynamic Facial Radiance Fields (DFRF) for few-shot talking head synthesis, which can rapidly generalize to an unseen identity with few training data. Different from the existing NeRF-based methods which directly encode the 3D geometry…
TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting
Beihang University（北京航空航天大学）；Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）；Griffith University（格里菲斯大学）；RIKEN AIP（理化学研究所人工智能项目）；The University of Tokyo（东京大学）
GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院，教育部高维信息智能感知与系统重点实验室，PCA实验室）；Tsientang Institute for Advanced Study（浙江大学前沿交叉研究院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
University of Science and Technology of China（中国科学技术大学）；Beijing Dilusense Technology Corporation（北京谛听视界科技有限公司）；Zhejiang University（浙江大学）；Tsinghua University（清华大学）
Generating high-fidelity talking head video by fitting with the input audio sequence is a challenging problem that receives considerable attentions recently. In this paper, we address this problem with the aid of neural scene representation networks. Our method is completely different from existing methods that rely on intermediate representations like 2D landmarks or 3D face models to bridge the gap between audio input and video output. Specifically, the feature of input audio signal is directly fed into a conditional implicit function to generate a dynamic neural radiance field, from which a…
EmbedTalk: Talking Head Synthesis using Gaussian Embeddings
School of Computer Science, University of Leeds（利兹大学计算机科学学院）；Leeds Institute of Medical Education, School of Medicine, University of Leeds（利兹大学医学教育研究所）
SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting
School of Information, Renmin University of China（中国人民大学信息学院）；School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与通信工程学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；School of Economics and Management, Tsinghua University（清华大学经济管理学院）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesis
National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学国家大数据系统计算技术工程实验室）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；University of Exeter（埃克塞特大学）
EmoGene: Audio-Driven Emotional 3D Talking-Head Generation
Khoury College of Computer Science, Northeastern University, USA（东北大学Khoury计算机科学学院）；Department of Electrical and Computer Engineering, Northeastern University, USA（东北大学电气与计算机工程系）
PASE: Phoneme-Aware Speech Encoder to Improve Lip Sync Accuracy for Talking Head Synthesis
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University（航空航天信息安全部分和可信计算重点实验室、教育部、网络安全科学与工程学院、武汉大学）；School of Cyber Science and Engineering, Wuhan University（网络安全科学与工程学院、武汉大学）；School of Police Information, Shandong Police College（警务信息学院、山东警察学院）
Learn2Talk: 3D Talking Face Learns from 2D Talking Face
School of Informatics, Xiamen University（厦门大学信息学院）；China Mobile (Hangzhou) Information Technology Co., Ltd.（中国移动（杭州）信息技术有限公司）；City University of Hong Kong（香港城市大学）
Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions
Engineering Sciences, Homi Bhabha National Institute Training School Complex（工程科学，霍米·巴赫瓦全国研究所培训学校）；Computer and Informatics Group, VECC 1/AF（计算机与信息组，VECC 1/AF）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Talking face synthesis has been widely studied in either appearance-based or warping-based methods. Previous works mostly utilize single face image as a source, and generate novel facial animations by merging other person's facial features. However, some facial regions like eyes or teeth, which may be hidden in the source image, can not be synthesized faithfully and stably. In this paper, We present a landmark driven two-stream network to generate faithful talking facial animation, in which more facial details are created, preserved and transferred from multiple source images instead of a sing…
SPEAK: Speech-Driven Pose and Emotion-Adjustable Talking Head Generation
Ping An Technology（平安科技）；Southeast University（东南大学）；Sun Yat-sen University（中山大学）；Nanjing University of Science and Technology（南京理工大学）；East China University of Science and Technology（华东理工大学）；University of Shanghai for Science and Technology（上海理工大学）
Renmin University of China（中国人民大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Psyche AI Inc.（Psyche AI公司）；Chinese Academy of Sciences（中国科学院）；Tsinghua University（清华大学）
Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. Traditional Generative Adversarial Networks (GAN) struggle to maintain consistent facial identity, while Neural Radiance Fields (NeRF) methods, although they can address this issue, often produce mismatched lip movements, inadequate facial expressions, and unstable head poses. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a fundamental flaw, …
FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motions
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
AIOZ；University of Liverpool（利物浦大学）；VinAI Research（VinAI研究院）
Audio-driven talking head animation is a challenging research topic with many real-world applications. Recent works have focused on creating photo-realistic 2D animation, while learning different talking or singing styles remains an open problem. In this paper, we present a new method to generate talking head animation with learnable style references. Given a set of style reference frames, our framework can reconstruct 2D talking head animation based on a single input image and an audio stream. Our method first produces facial landmarks motion from the audio stream and constructs the intermedi…
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2 School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China（1 中国科学院信息工程研究所，北京，中国 2 中国科学院大学网络空间安全学院，北京，中国）
One-shot talking head generation produces lip-sync talking heads based on arbitrary audio and one source face. To guarantee the naturalness and realness, recent methods propose to achieve free pose control instead of simply editing mouth areas. However, existing methods do not preserve accurate identity of source face when generating head motions. To solve the identity mismatch problem and achieve high-quality free pose control, we present One-shot Pose-controllable Talking head generation network (OPT). Specifically, the Audio Feature Disentanglement Module separates content features from aud…
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Beihang University（北京航空航天大学）；East China Normal University（华东师范大学）；Xiamen University（厦门大学）；Samsung Research America（三星美国研究院）
In this paper, we propose a talking face generation method that takes an audio signal as input and a short target video clip as reference, and synthesizes a photo-realistic video of the target face with natural lip motions, head poses, and eye blinks that are in-sync with the input audio signal. We note that the synthetic face attributes include not only explicit ones such as lip motions that have high correlations with speech, but also implicit ones such as head poses and eye blinks that have only weak correlation with the input audio. To model such complicated relationships among different f…
TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles
Tsinghua University（清华大学）；BNRist（北京信息科学与技术国家研究中心）；State Key Laboratory of Intelligent Technology and Systems（智能技术与系统国家重点实验室）；THUAI（清华大学人工智能国际研究院）；Fuxi AI Lab（伏羲人工智能实验室）；Netease（网易公司）；School of Computer Science, the University of Queensland（昆士兰大学计算机科学学院）
IIIT-Hyderabad（印度海得拉巴国际信息技术学院）；University of Bath（巴斯大学）
We leverage the modern advancements in talking head generation to propose an end-to-end system for talking head video compression. Our algorithm transmits pivot frames intermittently while the rest of the talking head video is generated by animating them. We use a state-of-the-art face reenactment network to detect key points in the non-pivot frames and transmit them to the receiver. A dense flow is then calculated to warp a pivot frame to reconstruct the non-pivot ones. Transmitting key points instead of full frames leads to significant compression. We propose a novel algorithm to adaptively …
We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables …
Netease Fuxi AI Lab（网易伏羲人工智能实验室）；University of Technology Sydney（悉尼科技大学）
In this paper, we propose a novel text-based talking-head video generation framework that synthesizes high-fidelity facial expressions and head motions in accordance with contextual sentiments as well as speech rhythm and pauses. To be specific, our framework consists of a speaker-independent stage and a speaker-specific stage. In the speaker-independent stage, we design three parallel networks to generate animation parameters of the mouth, upper face, and head from texts, separately. In the speaker-specific stage, we present a 3D face model guided attention network to synthesize videos tailor…
