发表机构
IIT Roorkee（印度理工学院罗尔基分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文指出Softpick在初始化时因分母归一化问题而失败，通过分离正负分母并采用停止梯度变体，在2.3亿参数下实现同等量化性能、更少死头及更可靠的passkey检索。
AI 中文摘要
Softmax注意力机制赋予每个token非零权重，在训练好的模型中，这些权重会集中形成注意力汇聚点（attention sinks）和巨大的激活值，从而拓宽了低精度推理必须覆盖的动态范围。Softpick通过修正分数来消除这一约束，去除注意力汇聚点并降低隐藏状态的峰度，但其优势在规模增大时逐渐消失。我们将这一失败重新定义为归一化问题。Softpick的分母分为正偏移和负偏移的和$D^+$和$D^-$，在前向和反向传播中相同使用，导致无法隔离它们各自的作用。我们将它们分离为一组算子，每个算子独立选择各自的分母。失败源于初始化阶段：每一层都存在$D^+$恰好为零的行，而接近死亡的行产生的梯度范数超过$10^{12}$，无论反向分母如何。只有Softpick和一种停止梯度变体（在前向中保持$D^+ + D^-$，但仅通过$D^+$进行反向传播）能够从头训练。在2.3亿参数规模下，停止梯度算子在量化性能上与Softpick相当，具有更少的死亡注意力头，并且更可靠地检索passkeys，仅在峰值注意力权重峰度上略逊一筹。
英文摘要
Softmax attention gives every token a nonzero weight, which in trained models concentrates into attention sinks and massive activations that widen the dynamic range low-precision inference must cover. Softpick removes this constraint by rectifying scores, eliminating sinks and lowering hidden-state kurtosis, but its advantage fades at scale. We reframe this failure as a normalization problem. Softpick's denominator splits into positive- and negative-shifted sums $D^+$ and $D^-$, used identically in the forward and backward pass, preventing their roles from being isolated. We separate them into a family of operators that independently choose each denominator. The failure originates at initialization: every layer contains rows where $D^+$ is exactly zero, while near-dead rows produce gradient norms above $10^{12}$ regardless of the backward denominator. Only Softpick and a stop-gradient variant, which keeps $D^+ + D^-$ forward but backpropagates through $D^+$ alone, train from scratch. At 230M parameters, the stop-gradient operator matches Softpick on quantization, has fewer dead heads, and retrieves passkeys more reliably, trailing only on peak attention-weight kurtosis.
Comments23 pages, 4 Figures
