从慢速双向到快速自回归视频扩散模型
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
AI总结：
本研究将双向视频扩散模型改编为自回归模型，结合分布匹配蒸馏与创新初始化及非对称蒸馏策略，实现单GPU上9.4 FPS的快速流式生成，并在VBench-Long上取得84.27分，超越所有先前模型。
摘要：
当前的视频扩散模型虽然生成了令人印象深刻的画质，但在交互式应用中却因双向注意力依赖而受限。生成单帧需要模型处理整个序列，包括未来帧。我们通过将预训练的双向扩散变换器改编为自回归变换器，实现了逐帧实时生成，从而解决了这一限制。为了进一步降低延迟，我们将分布匹配蒸馏（DMD）扩展到视频领域，将50步扩散模型蒸馏为4步生成器。为实现稳定且高质量的蒸馏，我们引入了基于教师模型ODE轨迹的学生初始化方案，以及一种非对称蒸馏策略，该策略使用双向教师模型来监督因果学生模型。这种方法有效缓解了自回归生成中的误差累积，使得尽管在短片段上训练，也能合成长时间视频。我们的模型在VBench-Long基准上取得了84.27的总分，超越了所有先前的视频生成模型。凭借KV缓存，我们的模型能在单个GPU上以9.4 FPS的速度实现高质量视频的快速流式生成。此外，我们的方法还支持零样本的流式视频到视频转换、图像到视频生成以及动态提示。
英文摘要：
Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models. It enables fast streaming generation of high-quality videos at 9.4 FPS on a single GPU thanks to KV caching. Our approach also enables streaming video-to-video translation, image-to-video, and dynamic prompting in a zero-shot manner.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
