DreamTalk：当情感说话人生成遇上扩散概率模型
DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models
AI总结：
DreamTalk利用扩散模型框架，通过去噪网络、风格感知唇部专家和风格预测器，实现跨多样化情感的高质量说话人生成，并支持从音频直接指定个性化情感。
摘要：
情感说话人生成已受到越来越多的关注。以往的方法主要基于GAN，在多样化的情感上仍难以持续产生令人满意的结果，并且无法方便地指定个性化情感。在本工作中，我们利用强大的扩散模型来解决这一问题，并提出了DreamTalk，这是一个通过精心设计来释放扩散模型在生成情感说话人方面潜力的框架。具体而言，DreamTalk由三个关键组件组成：去噪网络、风格感知唇部专家和风格预测器。基于扩散的去噪网络能够在多样化情感下持续合成高质量的音频驱动面部运动。为了提高唇部运动的准确性和情感饱满度，我们引入了一个风格感知唇部专家，它可以在保持情感强度的同时指导唇形同步。为了更方便地指定个性化情感，我们使用了一个基于扩散的风格预测器，直接从音频中预测个性化情感，从而无需额外的情感参考。通过这种方式，DreamTalk能够持续生成跨多样化情感的生动说话人脸，并方便地指定个性化情感。大量实验验证了DreamTalk的有效性和优越性。代码可在 https://github.com/ali-vilab/dreamtalk 获取。
英文摘要：
Emotional talking head generation has attracted growing attention. Previous methods, which are mainly GAN-based, still struggle to consistently produce satisfactory results across diverse emotions and cannot conveniently specify personalized emotions. In this work, we leverage powerful diffusion models to address the issue and propose DreamTalk, a framework that employs meticulous design to unlock the potential of diffusion models in generating emotional talking heads. Specifically, DreamTalk consists of three crucial components: a denoising network, a style-aware lip expert, and a style predictor. The diffusion-based denoising network can consistently synthesize high-quality audio-driven face motions across diverse emotions. To enhance lip-motion accuracy and emotional fullness, we introduce a style-aware lip expert that can guide lip-sync while preserving emotion intensity. To more conveniently specify personalized emotions, a diffusion-based style predictor is utilized to predict the personalized emotion directly from the audio, eliminating the need for extra emotion reference. By this means, DreamTalk can consistently generate vivid talking faces across diverse emotions and conveniently specify personalized emotions. Extensive experiments validate DreamTalk's effectiveness and superiority. The code is available at https://github.com/ali-vilab/dreamtalk.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
