DiffusionTalker：通过个性化器引导蒸馏实现高效紧凑的语音驱动3D数字人
DiffusionTalker: Efficient and Compact Speech-Driven 3D Talking Head via Personalizer-Guided Distillation
AI总结：
针对现有语音驱动3D面部动画方法缺乏个性化风格、效率与紧凑性不足的问题，提出DiffusionTalker，通过个性化器引导的蒸馏实现个性化动画生成，同时大幅提升推理速度并压缩模型体积，性能优于现有最优方法。
摘要：
实时语音驱动3D面部动画在学术界和工业界都颇具吸引力。传统方法主要聚焦于学习从语音到动画的确定性映射。近期的方法开始考虑语音驱动3D面部动画的非确定性特性，并将扩散模型应用于该任务。现有基于扩散的方法能够提升面部动画的多样性，但仍缺乏能传递准确唇语的个性化说话风格，此外，效率和紧凑性也有待提升。\n在本研究中，我们提出DiffusionTalker，通过个性化器引导的蒸馏来解决上述局限。在个性化方面，我们引入了对比个性化器（contrastive personalizer），用于学习身份和情感嵌入，从音频中捕捉说话风格。我们进一步在蒸馏过程中提出个性化增强器，以强化嵌入对面部动画的影响。在效率方面，我们采用迭代蒸馏来减少动画生成所需的步骤，实现了8倍以上的推理加速。为实现紧凑性，我们将大型教师模型蒸馏为更小的学生模型，在将性能损失降至最低的同时，使模型存储量减少了86.4%。蒸馏完成后，用户可以从音频中提取自身的身份和情感嵌入，快速生成能体现特定说话风格的个性化动画。大量实验表明，我们的方法优于当前最先进的方法。代码将发布于：https://github.com/ChenVoid/DiffusionTalker。
英文摘要：
Real-time speech-driven 3D facial animation has been attractive in academia and industry. Traditional methods mainly focus on learning a deterministic mapping from speech to animation. Recent approaches start to consider the nondeterministic fact of speech-driven 3D face animation and employ the diffusion model for the task. Existing diffusion-based methods can improve the diversity of facial animation. However, personalized speaking styles conveying accurate lip language is still lacking, besides, efficiency and compactness still need to be improved. In this work, we propose DiffusionTalker to address the above limitations via personalizer-guided distillation. In terms of personalization, we introduce a contrastive personalizer that learns identity and emotion embeddings to capture speaking styles from audio. We further propose a personalizer enhancer during distillation to enhance the influence of embeddings on facial animation. For efficiency, we use iterative distillation to reduce the steps required for animation generation and achieve more than 8x speedup in inference. To achieve compactness, we distill the large teacher model into a smaller student model, reducing our model's storage by 86.4\% while minimizing performance loss. After distillation, users can derive their identity and emotion embeddings from audio to quickly create personalized animations that reflect specific speaking styles. Extensive experiments are conducted to demonstrate that our method outperforms state-of-the-art methods. The code will be released at: https://github.com/ChenVoid/DiffusionTalker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
