通过混合注意力与解耦蒸馏实现长时域流式视频生成
Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation
AI总结：
本文提出混合强制方法，通过混合注意力设计联合优化时序信息保留与计算效率，采用轻量线性时序注意力和块稀疏注意力，实现高效稳定流式视频生成，实测在单块H100 GPU上达到29.5 FPS的实时832x480视频生成
摘要：
流式视频生成（SVG）通过将预训练的双向视频扩散模型蒸馏为具备滑动窗口注意力（SWA）的自回归模型。然而，SWA在长视频生成中不可避免地会丢失远距离历史，并且其计算开销仍是实时部署的关键挑战。本文提出混合强制方法，通过混合注意力设计联合优化时序信息保留与计算效率。首先，我们引入轻量线性时序注意力以保留滑动窗口之外的长距离依赖。特别地，我们维护一个紧凑的键值状态以逐步吸收被驱逐的token，以最小的内存和计算开销保留时序上下文。其次，我们将块稀疏注意力引入局部滑动窗口以减少短距离建模中的冗余计算，重新分配计算能力至更关键的依赖关系。最后，我们引入解耦蒸馏策略以适应混合注意力设计。在密集注意力下进行少量步骤的初始蒸馏，然后激活我们提出的线性时序和块稀疏注意力的蒸馏以进行流式建模，确保稳定优化。在短视频和长视频生成基准测试中，混合强制方法始终取得最先进的性能。值得注意的是，我们的模型在不进行量化或模型压缩的情况下，在单块NVIDIA H100 GPU上实现了实时、无限制的832x480视频生成，速度达到29.5 FPS。源代码和训练好的模型可在https://github.com/leeruibin/hybrid-forcing获取。
英文摘要：
Streaming video generation (SVG) distills a pretrained bidirectional video diffusion model into an autoregressive model equipped with sliding window attention (SWA). However, SWA inevitably loses distant history during long video generation, and its computational overhead remains a critical challenge to real-time deployment. In this work, we propose Hybrid Forcing, which jointly optimizes temporal information retention and computational efficiency through a hybrid attention design. First, we introduce lightweight linear temporal attention to preserve long-range dependencies beyond the sliding window. In particular, we maintain a compact key-value state to incrementally absorb evicted tokens, retaining temporal context with negligible memory and computational overhead. Second, we incorporate block-sparse attention into the local sliding window to reduce redundant computation within short-range modeling, reallocating computational capacity toward more critical dependencies. Finally, we introduce a decoupled distillation strategy tailored to the hybrid attention design. A few-step initial distillation is performed under dense attention, then the distillation of our proposed linear temporal and block-sparse attention is activated for streaming modeling, ensuring stable optimization. Extensive experiments on both short- and long-form video generation benchmarks demonstrate that Hybrid Forcing consistently achieves state-of-the-art performance. Notably, our model achieves real-time, unbounded 832x480 video generation at 29.5 FPS on a single NVIDIA H100 GPU without quantization or model compression. The source code and trained models are available at https://github.com/leeruibin/hybrid-forcing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
