GaussianTalker：基于3D Gaussian Splatting的说话人特定说话头部合成
GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting
AI总结：
针对NeRF音频驱动说话头部合成中唇动不同步、视觉抖动等问题，提出基于3D Gaussian Splatting的GaussianTalker，通过说话人特定运动转换与动态高斯渲染实现高质量实时合成。
摘要：
近期使用Neural Radiance Fields（NeRF，神经辐射场）进行音频驱动说话头部合成的工作取得了令人瞩目的成果。然而，由于NeRF隐式表示导致姿态和表情控制不足，这些方法仍存在一些局限，例如唇部动作不同步或不自然，以及视觉抖动和伪影。本文提出GaussianTalker，这是一种基于3D Gaussian Splatting的新型音频驱动说话头部合成方法。借助3D Gaussians的显式表示特性，该方法通过将Gaussians绑定到3D面部模型，实现对面部运动的直观控制。GaussianTalker由两个模块组成：Speaker-specific Motion Translator（说话人特定运动转换器）和Dynamic Gaussian Renderer（动态高斯渲染器）。Speaker-specific Motion Translator通过通用化音频特征提取和定制化唇部运动生成，实现针对目标说话人的准确唇部动作。Dynamic Gaussian Renderer引入Speaker-specific BlendShapes（说话人特定混合变形），通过潜在姿态增强面部细节表示，从而生成稳定且逼真的渲染视频。大量实验结果表明，GaussianTalker在说话头部合成方面优于现有最先进方法，可提供精确的唇音同步和出色的视觉质量。我们的方法在NVIDIA RTX4090 GPU上实现130 FPS的渲染速度，显著超过实时渲染性能阈值，并有潜力部署到其他硬件平台。
英文摘要：
Recent works on audio-driven talking head synthesis using Neural Radiance Fields (NeRF) have achieved impressive results. However, due to inadequate pose and expression control caused by NeRF implicit representation, these methods still have some limitations, such as unsynchronized or unnatural lip movements, and visual jitter and artifacts. In this paper, we propose GaussianTalker, a novel method for audio-driven talking head synthesis based on 3D Gaussian Splatting. With the explicit representation property of 3D Gaussians, intuitive control of the facial motion is achieved by binding Gaussians to 3D facial models. GaussianTalker consists of two modules, Speaker-specific Motion Translator and Dynamic Gaussian Renderer. Speaker-specific Motion Translator achieves accurate lip movements specific to the target speaker through universalized audio feature extraction and customized lip motion generation. Dynamic Gaussian Renderer introduces Speaker-specific BlendShapes to enhance facial detail representation via a latent pose, delivering stable and realistic rendered videos. Extensive experimental results suggest that GaussianTalker outperforms existing state-of-the-art methods in talking head synthesis, delivering precise lip synchronization and exceptional visual quality. Our method achieves rendering speeds of 130 FPS on NVIDIA RTX4090 GPU, significantly exceeding the threshold for real-time rendering performance, and can potentially be deployed on other hardware platforms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
