arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.08587cs.SDcs.AIeess.AS
收藏

EGSTalker:基于高效高斯形变的实时音频驱动说话头生成

EGSTalker: Real-Time Audio-Driven Talking Head Generation with Efficient Gaussian Deformation

Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center, School of Computer Science and Technology, Xinjiang University(新疆多模态智能处理与信息安全部门工程技术研究中心,计算机科学与技术学院,新疆大学);Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学);School of Electrical Engineering and Automation, Tianjin University of Technology(电气工程与自动化学院,天津理工大学)

Tianheng Zhu, Yinfeng Yu, Liejun Wang, Fuchun Sun, Wendong Zheng

AI总结:

EGSTalker是一种基于3D高斯泼溅的实时音频驱动说话头生成框架,通过静态高斯初始化和高效空间-音频注意力模块,在仅需3-5分钟训练视频的情况下,实现与最先进方法相当的渲染质量和唇形同步,并显著提升推理速度。

摘要:

本文提出了EGSTalker,一种基于3D高斯泼溅(3DGS)的实时音频驱动说话头生成框架。该框架旨在同时提升速度与视觉保真度,仅需3-5分钟的训练视频即可合成高质量的面部动画。框架包含两个关键阶段:静态高斯初始化与音频驱动形变。在第一阶段,使用多分辨率哈希三平面和Kolmogorov-Arnold网络(KAN)提取空间特征并构建紧凑的3D高斯表示。在第二阶段,我们提出了一种高效空间-音频注意力(ESAA)模块来融合音频与空间线索,同时KAN预测相应的高斯形变。大量实验表明,EGSTalker在渲染质量和唇形同步精度上与最先进方法相当,同时在推理速度上显著优于它们。这些结果凸显了EGSTalker在实时多媒体应用中的潜力。

英文摘要:

This paper presents EGSTalker, a real-time audio-driven talking head generation framework based on 3D Gaussian Splatting (3DGS). Designed to enhance both speed and visual fidelity, EGSTalker requires only 3-5 minutes of training video to synthesize high-quality facial animations. The framework comprises two key stages: static Gaussian initialization and audio-driven deformation. In the first stage, a multi-resolution hash triplane and a Kolmogorov-Arnold Network (KAN) are used to extract spatial features and construct a compact 3D Gaussian representation. In the second stage, we propose an Efficient Spatial-Audio Attention (ESAA) module to fuse audio and spatial cues, while KAN predicts the corresponding Gaussian deformations. Extensive experiments demonstrate that EGSTalker achieves rendering quality and lip-sync accuracy comparable to state-of-the-art methods, while significantly outperforming them in inference speed. These results highlight EGSTalker's potential for real-time multimedia applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑