快速自回归视频扩散与世界模型：基于时间缓存压缩与稀疏注意力
Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention
AI总结：
提出FAST-AR框架，通过TempCache压缩KV缓存、AnnCA加速交叉注意力、AnnSA稀疏化自注意力，实现自回归视频扩散模型5-10倍加速，同时保持视觉质量并稳定GPU内存使用。
摘要：
自回归视频扩散模型支持流式生成，为长序列合成、视频世界模型和交互式神经游戏引擎打开了大门。然而，其核心注意力层在推理时成为主要瓶颈：随着生成过程推进，KV缓存增长，导致延迟增加和GPU内存飙升，进而限制可用的时间上下文并损害长程一致性。在本工作中，我们研究了自回归视频扩散中的冗余性，并识别出三个持续存在的来源：跨帧的近似重复缓存键、缓慢演化的（主要是语义的）查询/键使得许多注意力计算冗余，以及长提示上的交叉注意力中每帧只有少量标记相关。基于这些观察，我们提出了一个统一的、无需训练的注意力框架（FAST-AR），用于快速自回归扩散，包含三个组件：TempCache通过时间对应压缩KV缓存以限制缓存增长；AnnCA通过使用快速近似最近邻（ANN）匹配选择帧相关的提示标记来加速交叉注意力；AnnSA通过将每个查询限制为语义匹配的键（也使用轻量级ANN）来稀疏化自注意力。这些模块共同减少了注意力、计算和内存，并且与现有的自回归扩散骨干网络和世界模型兼容。实验表明，在保持几乎相同的视觉质量的同时，实现了高达5-10倍的端到端加速，并且关键的是，在长序列生成中维持稳定的吞吐量和几乎恒定的峰值GPU内存使用，而先前的方法会逐渐变慢并遭受内存使用增加的问题。
英文摘要：
Autoregressive video diffusion models enable streaming generation, opening the door to long-form synthesis, video world models, and interactive neural game engines. However, their core attention layers become a major bottleneck at inference time: as generation progresses, the KV cache grows, causing both increasing latency and escalating GPU memory, which in turn restricts usable temporal context and harms long-range consistency. In this work, we study redundancy in autoregressive video diffusion and identify three persistent sources: near-duplicate cached keys across frames, slowly evolving (largely semantic) queries/keys that make many attention computations redundant, and cross-attention over long prompts where only a small subset of tokens matters per frame. Building on these observations, we propose a unified, training-free attention framework (FAST-AR) for FAST-AutoRegressive diffusion, consisting of three components: TempCache compresses the KV cache via temporal correspondence to bound cache growth; AnnCA accelerates cross-attention by selecting frame-relevant prompt tokens using fast approximate nearest neighbor (ANN) matching; and AnnSA sparsifies self-attention by restricting each query to semantically matched keys, also using a lightweight ANN. Together, these modules reduce attention, compute, and memory and are compatible with existing autoregressive diffusion backbones and world models. Experiments demonstrate up to x5 - x10 end-to-end speedups while preserving near-identical visual quality and, crucially, maintaining stable throughput and nearly constant peak GPU memory usage over long rollouts, where prior methods progressively slow down and suffer from increasing memory usage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
