注意力与残差汇流的统一视角：异常驱动的重缩放对变换器训练至关重要
A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
AI总结：
本文提出异常驱动重缩放对变换器训练的重要性，通过统一注意力与残差汇流的起源，展示了异常值在训练稳定性与性能提升中的关键作用。
摘要：
我们研究了大型语言模型中出现的异常值的功能作用，特别是注意力汇流（几个始终接收大注意力logits的token）和残差汇流（几个固定维度在大多数token上持续有大激活值）。我们假设这些异常值与相应的规范化（例如softmax注意力和RMSNorm）共同作用，有效地重缩放其他非异常值组件。我们将这种现象称为异常驱动重缩放，并在不同的模型架构和训练token数量下验证这一假设。这种观点统一了两种汇流类型的起源和缓解方法。我们的主要结论和观察包括：（1）异常值与规范化共同作用：移除规范化会消除相应的异常值，但会降低训练稳定性和性能；直接裁剪异常值而保留规范化会导致性能下降，表明异常驱动重缩放对训练稳定性有贡献。（2）异常值更多作为重缩放因子而非贡献者，因为最终的注意力和残差汇流的贡献显著小于非异常值。（3）异常值可以被吸收进可学习参数或通过显式门控重缩放来缓解，导致训练性能提高（平均提升2个百分点）和增强量化鲁棒性（在W4A4量化下下降1.2个百分点）
英文摘要：
We investigate the functional role of emergent outliers in large language models, specifically attention sinks (a few tokens that consistently receive large attention logits) and residual sinks (a few fixed dimensions with persistently large activations across most tokens). We hypothesize that these outliers, in conjunction with the corresponding normalizations (\textit{e.g.}, softmax attention and RMSNorm), effectively rescale other non-outlier components. We term this phenomenon \textit{outlier-driven rescaling} and validate this hypothesis across different model architectures and training token counts. This view unifies the origin and mitigation of both sink types. Our main conclusions and observations include: (1) Outliers function jointly with normalization: removing normalization eliminates the corresponding outliers but degrades training stability and performance; directly clipping outliers while retaining normalization leads to degradation, indicating that outlier-driven rescaling contributes to training stability. (2) Outliers serve more as rescale factors rather than contributors, as the final contributions of attention and residual sinks are significantly smaller than those of non-outliers. (3) Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling, leading to improved training performance (average gain of 2 points) and enhanced quantization robustness (1.2 points degradation under W4A4 quantization).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
