RT-Lynx：以正确方式将GEMM稀疏性应用于扩散模型
RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Models
AI总结：
针对扩散模型推理成本高的问题，提出将N:M半结构化稀疏性从权重转移到激活，结合误差补偿技术，实现线性层平均1.55倍加速且保持生成质量。
摘要：
扩散Transformer（DiT）在图像生成中表现出色，但推理成本高昂。先前工作通过量化和蒸馏降低了成本，但半结构化稀疏性（可将近减半FLOPs）仍未得到充分探索。一个关键原因是现有方法大多关注权重稀疏化，而剪枝50%的权重会移除关键模型容量并降低生成质量。然而，我们的研究表明，DiT激活本质上是稀疏的，并且比权重对N:M半结构化稀疏化更鲁棒。受此观察启发，我们倡导从权重稀疏化到激活稀疏化的范式转变。我们提出RT-Lynx，将N:M稀疏化应用于激活，并结合误差补偿技术以减轻精度损失。我们进一步针对此设置实现了高度优化的CUDA内核，在线性层中平均实现高达1.55倍的加速。在多个扩散模型上的大量实验表明，我们的方法在显著加速推理的同时，保持了原始模型的生成质量。
英文摘要：
Diffusion Transformers (DiT) achieve strong performance in image generation but incur substantial inference costs. While prior work has reduced this cost via quantization and distillation, semi-structured sparsity, which can nearly halve FLOPs, remains underexplored. A key reason is that most existing approaches focus on weight sparsification, and pruning 50% of the weights can remove critical model capacity and degrade generation quality. Our study, however, shows that DiT activations are intrinsically sparse and significantly more robust to N:M semi-structured sparsification than weights. Motivated by this observation, we advocate a paradigm shift from weight sparsification to activation sparsification. We propose RT-Lynx, which applies N:M sparsification to activations and incorporates error-compensation techniques to mitigate accuracy loss. We further implement highly optimized CUDA kernels tailored to this setting, achieving up to a 1.55x speedup on average in linear layers. Extensive experiments across multiple diffusion models demonstrate that our method preserves the generation quality of the original models while substantially accelerating inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
