基于参考引导深度压缩VAE的流式说话人肖像视频实时生成
Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs
AI总结：
提出一种结合因果视频VAE和自回归潜在去噪模型的流式说话人肖像视频生成框架，通过参考图像引导实现实时高质量生成。
摘要：
视频扩散模型显著推动了肖像视频生成的发展，但其高计算需求限制了在交互式应用中的使用。本文提出一个框架，用于生成以语音音频和参考图像为条件的可流式说话人肖像视频。该框架专为流式场景精心设计，包含一个用于深度潜在压缩的因果视频VAE和一个自回归潜在去噪模型。我们的因果VAE集成了可变数量的参考图像作为引导，使网络能够专注于动态信息而非静态外观，从而提升压缩效率和重建质量。此外，我们扩展了残差自编码范式，以改善VAE中的时空因果处理。生成器基于Rectified Flow Transformer架构，并以块状自回归方式生成视频潜在表示。我们的方法能够实时生成高质量的说话人肖像视频，速度显著快于基线模型。此外，综合实验表明，在逼真度、生动性和视频质量方面，该方法与这些大型模型相当甚至更优。
英文摘要：
Video diffusion models have significantly advanced portrait video generation, yet their high computational demands limit their use in interactive applications. This work presents a framework for streamable talking portrait video generation conditioned on speech audio and reference images. Designed meticulously for streaming scenarios, it features a causal video VAE for deep latent compression and an autoregressive latent denoising model. Our causal VAE integrates a variable number of reference images as guidance, allowing the network to focus on dynamic information rather than static appearance, thereby enhancing compression efficacy and reconstruction quality. Additionally, we extend the residual auto-encoding paradigm to improve spatial-temporal causality handling in our VAE. The generator is based on a Rectified Flow Transformer architecture and produces video latents in a blockwise auto-regressive manner. Our method enables the real-time generation of high-quality talking portrait videos, achieving speeds significantly faster than baseline models. Furthermore, comprehensive experiments demonstrate that it is on par with or even outperforms these large models in realism, vividness, and video quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
