REST：基于扩散模型的实时端到端流式说话人生成方法：通过ID上下文缓存和异步流式蒸馏
REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation
AI总结：
REST通过ID-Context缓存和异步流式蒸馏，实现高效实时端到端流式说话人生成。
摘要：
扩散模型在说话人生成（THG）领域显著推动了发展。然而，缓慢的推理速度和普遍的非自回归范式严重限制了基于扩散模型的THG模型的应用。在本研究中，我们提出REST，一种开创性的基于扩散模型、实时、端到端流式音频驱动说话人生成框架。为了支持实时端到端生成，首先通过具有高压缩比的时空变分自编码器学习紧凑的视频潜在空间。此外，为了在紧凑的视频潜在空间中实现半自回归流式生成，我们引入了ID-Context缓存机制，该机制将ID-Sink和Context-Cache原理整合到键值缓存中，以在长时间流式生成过程中保持身份一致性和时间一致性。此外，提出了一种异步流式蒸馏（ASD）策略，以减轻误差累积并增强流式生成中的时间一致性，利用非流式教师和异步噪声调度来监督流式学生。REST弥合了自回归和扩散方法之间的差距，实现了在需要实时THG应用中的效率突破。实验结果表明，REST在生成速度和总体性能上均优于最先进的方法。
英文摘要：
Diffusion models have significantly advanced the field of talking head generation (THG). However, slow inference speeds and prevalent non-autoregressive paradigms severely constrain the application of diffusion-based THG models. In this study, we propose REST, a pioneering diffusion-based, real-time, end-to-end streaming audio-driven talking head generation framework. To support real-time end-to-end generation, a compact video latent space is first learned through a spatiotemporal variational autoencoder with a high compression ratio. Additionally, to enable semi-autoregressive streaming within the compact video latent space, we introduce an ID-Context Cache mechanism, which integrates ID-Sink and Context-Cache principles into key-value caching for maintaining identity consistency and temporal coherence during long-term streaming generation. Furthermore, an Asynchronous Streaming Distillation (ASD) strategy is proposed to mitigate error accumulation and enhance temporal consistency in streaming generation, leveraging a non-streaming teacher with an asynchronous noise schedule to supervise the streaming student. REST bridges the gap between autoregressive and diffusion-based approaches, achieving a breakthrough in efficiency for applications requiring real-time THG. Experimental results demonstrate that REST outperforms state-of-the-art methods in both generation speed and overall performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
