发表机构
Graduate School of Mathematics, Nagoya University（名古屋大学大学院数学研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究证明高斯 softmax 注意力中，除 Perron 模式外，所有平方奇异值均收敛至体律上边缘，无离群值，并给出边缘的标量公式。
AI 中文摘要
我们研究行 softmax 自注意力机制，其中查询和键权重为独立高斯分布，处于比例区间，且逆温度固定。Hayase、Collins 和 Karakida 证明了在去除 Perron 方向后，经验平方奇异值分布的等价性。全局律本身并不排除有限多个非主导离群值。我们证明不存在此类离群值：对于每个固定的 k≥2，重新缩放的平方奇异值 ℓs_k(A)^2 依概率收敛到其体律的上边缘。事实上，这种收敛对于任何确定性的次线性数量的主导非 Perron 指数都是一致的。证明使用了 softmax 归一化的精确分解，对键矩阵施加条件，将条件协方差精确识别为对角共轭内积核，在算子范数下将该核线性化，并应用 Fan、Ma、Paquette 和 Wang 的支撑外局部律。一个单独的稳定性论证将有限条件变形 Marchenko-Pastur 边缘识别为极限体边缘。我们还推导了在整个比例区间内该边缘的标量公式，并恢复了 Hayase、Collins 和 Karakida 的显式平方模型公式，包括其物理分支。
英文摘要
We study row-softmax self-attention with independent Gaussian query and key weights in the proportional regime, at fixed inverse temperature. Hayase, Collins, and Karakida proved Gaussian equivalence for the empirical squared singular-value distribution after removal of the Perron direction. A global law alone does not exclude finitely many nonleading outliers. We prove that no such outliers persist: the rescaled squared singular value $\ell s_k(A)^2$ converges in probability to the upper edge of their bulk law for every fixed $k\ge 2$. In fact, this convergence is uniform over any deterministic sublinear number of leading non-Perron indices. The proof uses an exact decomposition of the softmax normalization, conditions on the key matrix, identifies the conditional covariance exactly with a diagonally conjugated inner-product kernel, linearizes that kernel in operator norm, and applies the outside-support local law of Fan, Ma, Paquette, and Wang. A separate stability argument identifies the finite conditional deformed Marchenko-Pastur edge with the limiting bulk edge. We also derive a scalar formula for that edge throughout the proportional regime and recover the explicit square-model formula of Hayase, Collins, and Karakida, including its physical branch.
