SparVAR：探索视觉自回归建模中的稀疏性以实现无训练加速
SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration
AI总结：
SparVAR通过利用自回归注意力的稀疏性，提出了一种无需训练的加速框架，有效降低计算复杂度，提升生成速度并保持图像质量。
摘要：
视觉自回归（VAR）建模因其创新的下一尺度预测范式而受到广泛关注。然而，主流VAR范式在每个自回归步骤中都处理所有历史尺度的token，随着下一尺度分辨率的增长，注意力的计算复杂度以四次方增长，导致显著延迟。先前的加速方法通常跳过高分辨率尺度，这虽然加快了推理速度，但丢弃了高频细节并损害了图像质量。为了解决这些问题，我们提出了SparVAR，一种无需训练的加速框架，利用了VAR注意力的三个特性：（i）强注意力sink，（ii）跨尺度激活相似性，以及（iii）明显的局部性。具体而言，我们动态地从稀疏决策尺度预测后续高分辨率尺度的稀疏注意力模式，并通过高效的索引映射机制构建尺度自相似的稀疏注意力，从而在大尺度上实现高效稀疏注意力计算。此外，我们提出了跨尺度局部稀疏注意力，并实现了高效的块状稀疏核，其前向速度比FlashAttention快超过5倍。大量实验表明，所提出的SparVAR能够将生成8B模型生产1024×1024高分辨率图像的时间减少到1秒，无需跳过最后一尺度。与通过FlashAttention加速的VAR基线相比，我们的方法在保持几乎所有高频细节的情况下实现了1.57倍的加速。当与现有的尺度跳过策略结合时，SparVAR可实现高达2.28倍的加速，同时保持具有竞争力的视觉生成质量。代码可在https://github.com/CAS-CLab/SparVAR上获得。
英文摘要：
Visual AutoRegressive (VAR) modeling has garnered significant attention for its innovative next-scale prediction paradigm. However, mainstream VAR paradigms attend to all tokens across historical scales at each autoregressive step. As the next scale resolution grows, the computational complexity of attention increases quartically with resolution, causing substantial latency. Prior accelerations often skip high-resolution scales, which speeds up inference but discards high-frequency details and harms image quality. To address these problems, we present \textbf{SparVAR}, a training-free acceleration framework that exploits three properties of VAR attention: \textbf{(i) strong attention sinks}, \textbf{(ii) cross-scale activation similarity}, and \textbf{(iii) pronounced locality}. Specifically, we dynamically predict the sparse attention pattern of later high-resolution scales from a sparse decision scale, and construct scale self-similar sparse attention via an efficient index-mapping mechanism, enabling high-efficiency sparse attention computation at large scales. Furthermore, we propose cross-scale local sparse attention and implement an efficient block-wise sparse kernel, which achieves $\mathbf{> 5\times}$ faster forward speed than FlashAttention. Extensive experiments demonstrate that the proposed SparVAR can reduce the generation time of an 8B model producing $1024\times1024$ high-resolution images to the \textbf{1s}, \textbf{without skipping the last scales}. Compared with the VAR baseline accelerated by FlashAttention, our method achieves a $\mathbf{1.57\times}$ speed-up while preserving almost all high-frequency details. When combined with existing scale-skipping strategies, SparVAR attains up to a $\mathbf{2.28\times}$ acceleration, while maintaining competitive visual generation quality. Code is available at \href{https://github.com/CAS-CLab/SparVAR}{SparVAR}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
