基于地标引导的扩散模型用于高保真且时间连贯的说话头像生成
Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation
AI总结：
本文提出一种两阶段扩散模型，先由语音生成同步面部地标，再以此为条件去噪，从而解决嘴部抖动问题，生成高保真且时间连贯的说话头像视频。
摘要：
音频驱动的说话头像生成是一项重要且具有挑战性的任务，适用于虚拟化身、电影制作和在线会议等多个领域。然而，现有的基于GAN的模型侧重于生成良好同步的唇形，但忽视了生成帧的视觉质量；而基于扩散的模型优先考虑生成高质量帧，却忽略了唇形匹配，导致嘴部运动抖动。为解决上述问题，我们提出了一种两阶段基于扩散的模型。第一阶段根据给定的语音生成同步的面部地标。在第二阶段，这些生成的地标作为去噪过程中的条件，旨在优化嘴部抖动问题，并生成高保真、良好同步且时间连贯的说话头像视频。大量实验表明，我们的模型取得了最佳性能。
英文摘要：
Audio-driven talking head generation is a significant and challenging task applicable to various fields such as virtual avatars, film production, and online conferences. However, the existing GAN-based models emphasize generating well-synchronized lip shapes but overlook the visual quality of generated frames, while diffusion-based models prioritize generating high-quality frames but neglect lip shape matching, resulting in jittery mouth movements. To address the aforementioned problems, we introduce a two-stage diffusion-based model. The first stage involves generating synchronized facial landmarks based on the given speech. In the second stage, these generated landmarks serve as a condition in the denoising process, aiming to optimize mouth jitter issues and generate high-fidelity, well-synchronized, and temporally coherent talking head videos. Extensive experiments demonstrate that our model yields the best performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
