DiffTalk：为泛化的音频驱动肖像动画构建扩散模型
DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
AI总结：
本文提出 DiffTalk，将音频驱动说话人头部生成建模为时间一致的潜在扩散去噪过程，并结合参考图像与 landmarks 条件，实现免微调的跨身份高质量、高分辨率同步动画。
摘要：
说话人头部合成是视频制作行业中一种很有前景的方法。近来，该研究领域已投入大量努力以提升生成质量或增强模型泛化能力。然而，很少有工作能够同时解决这两个问题，而这对实际应用至关重要。为此，本文将注意力转向新兴且强大的 Latent Diffusion Models，并将说话人头部生成建模为一个音频驱动、时间一致的去噪过程（DiffTalk）。更具体地说，我们没有将音频信号用作唯一驱动因素，而是研究说话面部的控制机制，并将参考面部图像和 landmarks（关键点）作为条件纳入，以实现具有个性感知的泛化合成。通过这种方式，所提出的 DiffTalk 能够生成与源音频同步的高质量说话人头部视频；更重要的是，它无需任何进一步微调即可自然泛化到不同身份。此外，DiffTalk 还能以可忽略的额外计算成本被灵活调整，用于更高分辨率合成。大量实验表明，所提出的 DiffTalk 可为泛化的新身份高效合成高保真音频驱动说话人头部视频。更多视频结果请参见 https://sstzal.github.io/DiffTalk/。
英文摘要：
Talking head synthesis is a promising approach for the video production industry. Recently, a lot of effort has been devoted in this research area to improve the generation quality or enhance the model generalization. However, there are few works able to address both issues simultaneously, which is essential for practical applications. To this end, in this paper, we turn attention to the emerging powerful Latent Diffusion Models, and model the Talking head generation as an audio-driven temporally coherent denoising process (DiffTalk). More specifically, instead of employing audio signals as the single driving factor, we investigate the control mechanism of the talking face, and incorporate reference face images and landmarks as conditions for personality-aware generalized synthesis. In this way, the proposed DiffTalk is capable of producing high-quality talking head videos in synchronization with the source audio, and more importantly, it can be naturally generalized across different identities without any further fine-tuning. Additionally, our DiffTalk can be gracefully tailored for higher-resolution synthesis with negligible extra computational cost. Extensive experiments show that the proposed DiffTalk efficiently synthesizes high-fidelity audio-driven talking head videos for generalized novel identities. For more video results, please refer to \url{https://sstzal.github.io/DiffTalk/}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
