arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18849cs.LGcs.AI
收藏

变换器稳定性与注意力敏感性的精确性

Exact Attention Sensitivity and the Geometry of Transformer Stability

UNC-Chapel Hill, Kenan-Flagler Business School, Chapel Hill, NC, USA(北卡罗来纳大学教堂山分校,肯纳-弗拉格勒商学院)

Seyed Morteza Emadi

AI总结:

本文通过推导softmax雅可比矩阵的精确算子范数,揭示了变换器稳定性源于架构梯度流动,而非注意力动态,并解释了预LN、DeepNorm缩放和预热等机制的原理。

摘要:

尽管驱动着现代人工智能,变换器仍然神秘地对训练敏感。我们开发了一种稳定性理论,解释了为什么预层归一化有效,为什么DeepNorm使用N^{-1/4}的缩放,以及为什么需要预热,所有这些都基于第一原理。我们的框架有两个支柱:(1)我们推导出softmax雅可比矩阵的精确算子范数,||J_softmax(u/τ)||_{∞→1}=θ(p)/τ,其中平衡质量因子θ(p)∈[0,1]量化了注意力敏感性。(2)我们引入了块-∞/RMS几何,与逐token计算对齐,从而得到与序列长度无关的利普希茨界。利用这一框架,我们证明了预LN保持身份梯度路径,而后LN的层归一化雅可比矩阵随着深度呈指数级放大,我们还显示DeepNorm的N^{-1/4}源于注意力四个投影矩阵的四次结构。我们验证了我们的理论在774M参数模型上的有效性,并发现,与直觉相反,注意力在训练过程中变锐以减少敏感性,θ(p)≈1在整个过程中保持不变。变换器的稳定性完全来自于架构的梯度流动,而不是注意力动态。这一发现改变了我们对训练的思考:架构本身必须处理敏感性,而不是学习的注意力模式。

英文摘要:

We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation. Our main result is the exact identity $\|J_τ(u)\|_{\infty\to1}=θ(p)/τ$ for the Jacobian $J_τ(u)$ of the tempered softmax $u\mapsto\mathrm{softmax}(u/τ)$, where $θ(p)=4\max_{S\subseteq[L]}p(S)(1-p(S))$ measures how evenly the attention distribution can be bisected rather than how concentrated it is. We combine this identity with a block-$\infty$/RMS norm under which row-stochastic attention mixing is nonexpansive. This yields a distribution-aware local Jacobian bound for multi-head attention and a sequence-length-independent Lipschitz bound on bounded input sets, with explicit dependence on width, input magnitude, temperature, and projection norms. We also identify a structural distinction between normalization placements: a pre-LN residual-sublayer Jacobian contains an additive identity term, whereas a post-LN residual-sublayer Jacobian does not. A LayerNorm projection lemma gives a sufficient condition under which the LayerNorm-only term in the post-LN expansion contracts geometrically; the condition is not tested by our experiments. Across three Pre-LN early-training runs of $774$M-parameter models, attention becomes substantially more concentrated while the median lower-bound certificate for $θ(p)$ remains near one at every sampled layer and checkpoint. This certifies near-maximal exact sensitivity for at least half of the sampled rows within each layer. A minority of rows enters a dominant-atom regime with lower exact sensitivity, consistent with the deterministic relationship between $p_{\max}$ and $θ(p)$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑