DiffusionTalker：面向语音驱动3D人脸扩散模型的个性化与加速
DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser
AI总结：
DiffusionTalker提出基于扩散模型的语音驱动3D人脸动画方法，通过对比学习实现个性化说话风格，并利用知识蒸馏将生成步数从数百步压缩至8步以加速推理，实验表明其性能优于现有最先进方法。
摘要：
语音驱动的3D人脸动画在学术界和工业界一直是一项具有吸引力的任务。传统方法主要关注学习从语音到动画的确定性映射。近期方法开始考虑语音驱动3D人脸动画的非确定性事实，并采用扩散模型来完成该任务。然而，个性化人脸动画和加速动画生成仍然是现有基于扩散方法的两大主要局限。为解决上述局限，我们提出DiffusionTalker，一种基于扩散的方法，利用对比学习实现3D人脸动画个性化，并利用知识蒸馏加速3D动画生成。具体而言，为实现个性化，我们引入一个可学习的说话身份来聚合音频序列中的知识。所提出的身份嵌入以对比学习方式提取不同人物之间的定制化面部线索。在推理过程中，用户可以根据输入音频获得个性化的人脸动画，反映特定的说话风格。对于具有数百步的已训练扩散模型，我们将其蒸馏为一个仅需8步的轻量级模型以实现加速。大量实验表明，我们的方法优于最先进的方法。代码将公开发布。
英文摘要：
Speech-driven 3D facial animation has been an attractive task in both academia and industry. Traditional methods mostly focus on learning a deterministic mapping from speech to animation. Recent approaches start to consider the non-deterministic fact of speech-driven 3D face animation and employ the diffusion model for the task. However, personalizing facial animation and accelerating animation generation are still two major limitations of existing diffusion-based methods. To address the above limitations, we propose DiffusionTalker, a diffusion-based method that utilizes contrastive learning to personalize 3D facial animation and knowledge distillation to accelerate 3D animation generation. Specifically, to enable personalization, we introduce a learnable talking identity to aggregate knowledge in audio sequences. The proposed identity embeddings extract customized facial cues across different people in a contrastive learning manner. During inference, users can obtain personalized facial animation based on input audio, reflecting a specific talking style. With a trained diffusion model with hundreds of steps, we distill it into a lightweight model with 8 steps for acceleration. Extensive experiments are conducted to demonstrate that our method outperforms state-of-the-art methods. The code will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
