arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.21079cs.CV
收藏

免训练的自适应稀疏注意力用于高效长视频生成

Training-free and Adaptive Sparse Attention for Efficient Long Video Generation

Peking University（北京大学）；ByteDance（字节跳动）

Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, Bin Cui

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出AdaSpa，一种免训练的即插即用稀疏注意力方法，通过动态块化模式与在线精确搜索，显著加速扩散变换器长视频生成，同时保持视频质量。

摘要：

使用扩散变换器（DiTs）生成高保真长视频通常受到显著延迟的阻碍，这主要是由于注意力机制的计算需求所致。例如，使用HunyuanVideo生成一个8秒720p视频（110K个令牌）需要约600 PFLOPs，其中约500 PFLOPs消耗在注意力计算上。为解决此问题，我们提出了AdaSpa，这是首个动态模式与在线精确搜索的稀疏注意力方法。首先，为实现动态模式，我们引入了一种块化模式，以高效捕获DiTs中固有的层次稀疏性。这基于我们的观察：DiTs的稀疏特性在不同模态之间及内部表现出层次化和块化结构。这种块化方法显著降低了注意力计算的复杂度，同时保持了生成视频的高保真度。其次，为实现在线精确搜索，我们提出了融合LSE缓存搜索与头自适应层次块稀疏注意力。该方法受我们发现的启发：DiTs的稀疏模式和LSE随输入、层和头而变化，但在去噪步骤间保持不变。通过利用这种去噪步骤间的不变性，它适应了DiTs的动态特性，并允许以最小开销精确、实时地识别稀疏索引。AdaSpa作为一种自适应的即插即用解决方案实现，可无缝集成到现有DiTs中，既不需要额外的微调，也不需要依赖数据集的性能分析。大量实验验证了AdaSpa在各种模型上提供显著加速的同时保持视频质量，使其成为高效视频生成的稳健且可扩展的方法。

英文摘要：

Generating high-fidelity long videos with Diffusion Transformers (DiTs) is often hindered by significant latency, primarily due to the computational demands of attention mechanisms. For instance, generating an 8-second 720p video (110K tokens) with HunyuanVideo takes about 600 PFLOPs, with around 500 PFLOPs consumed by attention computations. To address this issue, we propose AdaSpa, the first Dynamic Pattern and Online Precise Search sparse attention method. Firstly, to realize the Dynamic Pattern, we introduce a blockified pattern to efficiently capture the hierarchical sparsity inherent in DiTs. This is based on our observation that sparse characteristics of DiTs exhibit hierarchical and blockified structures between and within different modalities. This blockified approach significantly reduces the complexity of attention computation while maintaining high fidelity in the generated videos. Secondly, to enable Online Precise Search, we propose the Fused LSE-Cached Search with Head-adaptive Hierarchical Block Sparse Attention. This method is motivated by our finding that DiTs' sparse pattern and LSE vary w.r.t. inputs, layers, and heads, but remain invariant across denoising steps. By leveraging this invariance across denoising steps, it adapts to the dynamic nature of DiTs and allows for precise, real-time identification of sparse indices with minimal overhead. AdaSpa is implemented as an adaptive, plug-and-play solution and can be integrated seamlessly with existing DiTs, requiring neither additional fine-tuning nor a dataset-dependent profiling. Extensive experiments validate that AdaSpa delivers substantial acceleration across various models while preserving video quality, establishing itself as a robust and scalable approach to efficient video generation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑