READ：用于音频驱动说话头生成的实时高效异步扩散
READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation
AI总结：
针对基于扩散的音频驱动说话头生成模型推理速度慢的问题，提出READ框架，通过时间VAE压缩视频潜在空间、SpeechAE生成对齐语音潜在编码、A2V-DiT建模及异步噪声调度器，实现质量与速度的最优平衡。
摘要：
扩散模型的引入为音频驱动说话头生成领域带来了重大进展。然而，极慢的推理速度严重限制了基于扩散的说话头生成模型的实际应用。在本研究中，我们提出READ，一种基于实时扩散Transformer的说话头生成框架。我们的方法首先通过时间变分自动编码器（temporal VAE）学习时空高度压缩的视频潜在空间，显著减少token数量以加速生成。为了在该压缩潜在空间内实现更好的音视频对齐，我们提出预训练的语音自动编码器（SpeechAE）来生成与视频潜在空间对应的时间压缩语音潜在编码。这些潜在表示随后由精心设计的音频到视频扩散Transformer（A2V-DiT）主干进行建模，以实现高效的说话头合成。此外，为了在扩展生成中确保时间一致性和加速推理，我们为框架的训练和推理过程提出了一种新颖的异步噪声调度器（ANS）。ANS在潜在空间中利用异步加噪和异步运动引导生成，确保生成视频片段的一致性。实验结果表明，READ通过生成具有竞争力的说话头视频且运行时间显著减少，优于最先进的方法，在质量和速度之间实现了最佳平衡，同时在长时间生成中保持了稳健的指标稳定性。
英文摘要：
The introduction of diffusion models has brought significant advances to the field of audio-driven talking head generation. However, the extremely slow inference speed severely limits the practical implementation of diffusion-based talking head generation models. In this study, we propose READ, a real-time diffusion-transformer-based talking head generation framework. Our approach first learns a spatiotemporal highly compressed video latent space via a temporal VAE, significantly reducing the token count to accelerate generation. To achieve better audio-visual alignment within this compressed latent space, a pre-trained Speech Autoencoder (SpeechAE) is proposed to generate temporally compressed speech latent codes corresponding to the video latent space. These latent representations are then modeled by a carefully designed Audio-to-Video Diffusion Transformer (A2V-DiT) backbone for efficient talking head synthesis. Furthermore, to ensure temporal consistency and accelerated inference in extended generation, we propose a novel asynchronous noise scheduler (ANS) for both the training and inference processes of our framework. The ANS leverages asynchronous add-noise and asynchronous motion-guided generation in the latent space, ensuring consistency in generated video clips. Experimental results demonstrate that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime, achieving an optimal balance between quality and speed while maintaining robust metric stability in long-time generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
