EGSTalker: Real-Time Audio-Driven Talking Head Generation with Efficient Gaussian Deformation
Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center, School of Computer Science and Technology, Xinjiang University(新疆多模态智能处理与信息安全部门工程技术研究中心,计算机科学与技术学院,新疆大学);Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学);School of Electrical Engineering and Automation, Tianjin University of Technology(电气工程与自动化学院,天津理工大学)
This paper presents EGSTalker, a real-time audio-driven talking head generation framework based on 3D Gaussian Splatting (3DGS). Designed to enhance both speed and visual fidelity, EGSTalker requires only 3-5 minutes of training video to synthesize high-quality facial animations. The framework comprises two key stages: static Gaussian initialization and audio-driven deformation. In the first stage, a multi-resolution hash triplane and a Kolmogorov-Arnold Network (KAN) are used to extract spatial features and construct a compact 3D Gaussian representation. In the second stage, we propose an Efficient Spatial-Audio Attention (ESAA) module to fuse audio and spatial cues, while KAN predicts the corresponding Gaussian deformations. Extensive experiments demonstrate that EGSTalker achieves rendering quality and lip-sync accuracy comparable to state-of-the-art methods, while significantly outperforming them in inference speed. These results highlight EGSTalker's potential for real-time multimedia applications.
DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations
Renmin University of China(中国人民大学);Ant Group(蚂蚁集团);Tsinghua University(清华大学);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算创新中心)
LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild
University of Science and Technology Beijing(北京科技大学);Xiaoduo Intelligent Technology (Beijing) Co., Ltd(小多智能技术(北京)有限公司);Department of Computer Science, Changzhi University(长治大学计算机科学系)
PGSTalker: Real-Time Audio-Driven Talking Head Generation via 3D Gaussian Splatting with Pixel-Aware Density Control
Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center(新疆多模态智能处理与信息安全工程技术创新研究中心);School of Computer Science and Technology, Xinjiang University(新疆大学计算机科学与技术学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);School of Electrical Engineering and Automation, Tianjin University of Technology(天津理工大学电气工程与自动化学院)
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Alibaba Group(阿里巴巴集团);Nanjing University(南京大学)
Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation
ReLER, CCAI, Zhejiang University(ReLER,中国人工智能学会,浙江大学);School of Cyber Science and Technology, Sun Yat-sen University(信息科学与技术学院,孙中山大学);Department of Statistics, University of Wisconsin–Madison(统计学系,威斯康星大学麦迪逊分校)
Audio-Plane: Audio Factorization Plane Gaussian Splatting for Real-Time Talking Head Synthesis
School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电子与电气工程学院);John A. Paulson School of Engineering and Applied Sciences at Harvard University(哈佛大学约翰·A·保罗森工程与应用科学学院);PhiGent Robotics(PhiGent机器人技术公司);Beijing National Research Center for Information Science and Technology (BNRist) and the Department of Automation, Tsinghua University(北京信息科学与技术国家研究中心(BNRist)和清华大学自动化系)