高效的自回归视频扩散模型与Dummy头
Efficient Autoregressive Video Diffusion with Dummy Head
AI总结：
本文提出Dummy Forcing方法，通过优化多头自注意力机制的内存分配和上下文管理，提升视频扩散模型的生成速度，同时保持高质量输出。
摘要：
自回归视频扩散模型由于其因果建模和迭代去噪特性，最近引起了广泛的研究兴趣。在本工作中，我们发现这些模型中的多头自注意力机制对历史帧的利用不足：大约25%的头几乎只关注当前帧，丢弃它们的KV缓存只会导致轻微的性能下降。在此基础上，我们提出了Dummy Forcing，一种简单而有效的控制不同头之间上下文可访问性的方法。具体而言，所提出的异构内存分配减少了头间的上下文冗余，同时动态头编程用于自适应分类头类型。此外，我们开发了一种上下文打包技术以实现更激进的缓存压缩。无需额外训练，我们的Dummy Forcing在基线模型上实现了高达2.0倍的速度提升，支持在24.3 FPS下进行视频生成，质量下降不到0.5%。项目页面可在https://csguoh.github.io/project/DummyForcing/上获取。
英文摘要：
The autoregressive video diffusion model has recently gained considerable research interest due to its causal modeling and iterative denoising. In this work, we identify that the multi-head self-attention in these models under-utilizes historical frames: approximately 25% heads attend almost exclusively to the current frame, and discarding their KV caches incurs only minor performance degradation. Building upon this, we propose Dummy Forcing, a simple yet effective method to control context accessibility across different heads. Specifically, the proposed heterogeneous memory allocation reduces head-wise context redundancy, accompanied by dynamic head programming to adaptively classify head types. Moreover, we develop a context packing technique to achieve more aggressive cache compression. Without additional training, our Dummy Forcing delivers up to 2.0x speedup over the baseline, supporting video generation at 24.3 FPS with less than 0.5% quality drop. Project page is available at https://csguoh.github.io/project/DummyForcing/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
