面向扩散语言模型的Sink-aware剪枝
Sink-Aware Pruning for Diffusion Language Models
AI总结：
面向扩散语言模型的Sink-aware剪枝方法通过识别并剪枝不稳定sink，提升模型效率与质量的平衡，优于现有基线方法。
摘要：
扩散语言模型（DLMs）由于迭代去噪过程导致推理成本较高，这促使了高效的剪枝方法的发展。现有的剪枝启发式方法大多继承自自回归（AR）语言模型，通常保留注意力sink token，因为AR中的sink充当稳定的全局锚点。我们证明这一假设在DLMs中不成立：注意力sink位置在整个生成轨迹中表现出显著的方差（通过主导sink位置在时间步中的变化程度来衡量），表明sink通常是短暂的且在结构上不如AR模型重要。基于这一观察，我们提出了${\bf \texttt{Sink-Aware Pruning}}$，该方法自动识别并剪枝DLMs中的不稳定sink（先前研究通常为AR LLMs保留sink）。无需重新训练，我们的方法在质量-效率权衡上表现更优，并在匹配的计算资源下优于强大的前期剪枝基线。我们的代码可在https://github.com/VILA-Lab/Sink-Aware-Pruning上获得。
英文摘要：
Diffusion Language Models (DLMs) incur high inference cost due to iterative denoising, motivating efficient pruning. Existing pruning heuristics largely inherited from autoregressive (AR) LLMs, typically preserve attention sink tokens because AR sinks serve as stable global anchors. We show that this assumption does not hold for DLMs: the attention-sink position exhibits substantially higher variance over the full generation trajectory (measured by how the dominant sink locations shift across timesteps), indicating that sinks are often transient and less structurally essential than in AR models. Based on this observation, we propose ${\bf \texttt{Sink-Aware Pruning}}$, which automatically identifies and prunes unstable sinks in DLMs (prior studies usually keep sinks for AR LLMs). Without retraining, our method achieves a better quality-efficiency trade-off and outperforms strong prior pruning baselines under matched compute. Our code is available at https://github.com/VILA-Lab/Sink-Aware-Pruning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
