D^3-Talker：用于少样本3D说话人头合成的双分支解耦变形场
D^3-Talker: Dual-Branch Decoupled Deformation Fields for Few-Shot 3D Talking Head Synthesis
AI总结：
针对少样本3D说话人头合成中音频无关信息导致的唇部同步差和质量低问题，提出D^3-Talker，通过双分支解耦变形场及相似性对比损失、从粗到细模块，实现更优的高保真渲染与音频-唇部同步。
摘要：
3D说话人头合成的关键挑战在于，为每个目标身份从头训练新模型时需依赖长时长说话人头视频。近期方法尝试通过预训练模型从音频中提取通用特征以解决该问题。然而，由于音频包含与唇部运动无关的信息，现有方法在仅用少量帧训练时，通常难以将给定音频映射到目标人脸的真实唇部行为，导致唇部同步性差且说话人头图像质量低。本文提出D^3-Talker，一种构建静态3D高斯属性场并利用音频和面部运动信号独立控制两个不同高斯属性变形场的新方法，有效解耦通用和个性化变形的预测。我们在预训练期间设计一种新颖的相似性对比损失函数以实现更彻底的解耦。此外，我们集成一个从粗到细模块来优化渲染图像，缓解头部运动导致的模糊并提升整体图像质量。大量实验表明，在有限训练数据下，D^3-Talker在高保真渲染和准确的音频-唇部同步方面优于现有方法。我们的代码将在论文被接收后提供。
英文摘要：
A key challenge in 3D talking head synthesis lies in the reliance on a long-duration talking head video to train a new model for each target identity from scratch. Recent methods have attempted to address this issue by extracting general features from audio through pre-training models. However, since audio contains information irrelevant to lip motion, existing approaches typically struggle to map the given audio to realistic lip behaviors in the target face when trained on only a few frames, causing poor lip synchronization and talking head image quality. This paper proposes D^3-Talker, a novel approach that constructs a static 3D Gaussian attribute field and employs audio and Facial Motion signals to independently control two distinct Gaussian attribute deformation fields, effectively decoupling the predictions of general and personalized deformations. We design a novel similarity contrastive loss function during pre-training to achieve more thorough decoupling. Furthermore, we integrate a Coarse-to-Fine module to refine the rendered images, alleviating blurriness caused by head movements and enhancing overall image quality. Extensive experiments demonstrate that D^3-Talker outperforms state-of-the-art methods in both high-fidelity rendering and accurate audio-lip synchronization with limited training data. Our code will be provided upon acceptance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
