DAWN：用于说话头视频生成的非自回归扩散框架的动态帧化身
DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
AI总结：
针对自回归说话头生成方法的上下文利用受限、误差累积和速度慢问题，提出非自回归扩散框架DAWN，一次性生成动态长度视频，实现精确唇动与自然姿态，支持高质量长视频。代码已开源。
摘要：
说话头生成旨在从单张肖像和语音音频片段中生成生动逼真的说话头视频。尽管基于扩散的说话头生成已取得显著进展，但几乎所有方法都依赖自回归策略，这些策略存在超出当前生成步骤的上下文利用受限、错误累积和生成速度较慢等问题。为应对这些挑战，我们提出了DAWN（动态帧化身与非自回归扩散），一个能够一次性生成动态长度视频序列的框架。具体而言，它包含两个主要组件：（1）在潜在运动空间中基于音频驱动的整体面部动态生成，以及（2）基于音频驱动的头部姿态和眨眼生成。大量实验表明，我们的方法能生成具有精确唇部运动、自然姿态/眨眼动作的真实生动视频。此外，凭借高生成速度，DAWN具备强大的外推能力，确保高质量长视频的稳定产出。这些结果凸显了DAWN在说话头视频生成领域的巨大潜力和影响。进一步，我们希望DAWN能激发扩散模型中非自回归方法的更多探索。我们的代码将在https://github.com/Hanbo-Cheng/DAWN-pytorch公开提供。
英文摘要：
Talking head generation intends to produce vivid and realistic talking head videos from a single portrait and speech audio clip. Although significant progress has been made in diffusion-based talking head generation, almost all methods rely on autoregressive strategies, which suffer from limited context utilization beyond the current generation step, error accumulation, and slower generation speed. To address these challenges, we present DAWN (Dynamic frame Avatar With Non-autoregressive diffusion), a framework that enables all-at-once generation of dynamic-length video sequences. Specifically, it consists of two main components: (1) audio-driven holistic facial dynamics generation in the latent motion space, and (2) audio-driven head pose and blink generation. Extensive experiments demonstrate that our method generates authentic and vivid videos with precise lip motions, and natural pose/blink movements. Additionally, with a high generation speed, DAWN possesses strong extrapolation capabilities, ensuring the stable production of high-quality long videos. These results highlight the considerable promise and potential impact of DAWN in the field of talking head video generation. Furthermore, we hope that DAWN sparks further exploration of non-autoregressive approaches in diffusion models. Our code will be publicly available at https://github.com/Hanbo-Cheng/DAWN-pytorch.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
