arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01077cs.CV
收藏

PISA:分块稀疏注意力使扩散变换器更高效

PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers

The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))

Haopeng Li, Shitong Shao, Wenliang Zhong, Zikai Zhou, Lichen Bai, Hui Xiong, Zeke Xie

AI总结:

PISA通过分块稀疏注意力在保持质量的同时显著提升扩散变换器的效率。

摘要:

扩散变换器在视频和图像生成中至关重要,但其效率受到注意力二次复杂度的限制。尽管块稀疏注意力通过仅关注关键键值块来加速计算,但在高稀疏度时会因丢弃上下文而退化。在本工作中,我们发现非关键块的注意力分数具有分布稳定性,允许其被准确且高效地近似而非丢弃,这对稀疏注意力设计至关重要。受这一关键洞察启发,我们提出了PISA,一种无需训练的分块稀疏注意力,其复杂度为次二次,能够覆盖完整的注意力范围。与传统保留或丢弃范式直接丢弃非关键块信息不同,PISA引入了一种新的精确或近似策略:它对关键块保持精确计算,通过块级泰勒展开高效近似其余部分。这种设计使PISA能够作为完整注意力的忠实代理,有效弥合速度与质量之间的差距。实验结果表明,PISA在Wan2.1-14B和Hunyuan-Video上分别实现了1.91倍和2.57倍的速度提升,同时在稀疏注意力方法中始终保持最高质量。值得注意的是,即使在FLUX图像生成上,PISA也能在不牺牲视觉质量的情况下实现1.2倍的加速。代码可在:https://github.com/xie-lab-ml/piecewise-sparse-attention获取。

英文摘要:

Diffusion Transformers are fundamental for video and image generation, but their efficiency is bottlenecked by the quadratic complexity of attention. While block sparse attention accelerates computation by attending only critical key-value blocks, it suffers from degradation at high sparsity by discarding context. In this work, we discover that attention scores of non-critical blocks exhibit distributional stability, allowing them to be approximated accurately and efficiently rather than discarded, which is essentially important for sparse attention design. Motivated by this key insight, we propose PISA, a training-free Piecewise Sparse Attention that covers the full attention span with sub-quadratic complexity. Unlike the conventional keep-or-drop paradigm that directly drop the non-critical block information, PISA introduces a novel exact-or-approximate strategy: it maintains exact computation for critical blocks while efficiently approximating the remainder through block-wise Taylor expansion. This design allows PISA to serve as a faithful proxy to full attention, effectively bridging the gap between speed and quality. Experimental results demonstrate that PISA achieves 1.91 times and 2.57 times speedups on Wan2.1-14B and Hunyuan-Video, respectively, while consistently maintaining the highest quality among sparse attention methods. Notably, even for image generation on FLUX, PISA achieves a 1.2 times acceleration without compromising visual quality. Code is available at: https://github.com/xie-lab-ml/piecewise-sparse-attention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑