发表机构
University of Exeter（埃克塞特大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出牛顿-舒尔茨注意力（NS-Attn.），一种无参数变换，通过降低注意力输出矩阵的谱集中度并增加有效秩，在ViT和Swin上于CIFAR-10/100的12个匹配种子比较中平均提升0.25-0.83个百分点准确率，但增加推理延迟。
AI 中文摘要
牛顿-舒尔茨（NS）迭代最近被用于Muon优化器中，以在大型语言模型训练期间变换更新矩阵。受其谱效应的启发，我们研究将NS直接应用于Transformer的注意力表示。我们引入了牛顿-舒尔茨注意力（NS-Attn.），这是一种应用于每个注意力头输出的无参数变换。每个头的输出被排列为特征×令牌矩阵，并通过其Frobenius范数进行归一化。然后，我们应用一个有限的NS多项式步骤并恢复原始范数。其目标是降低谱集中度并在标准头合并和输出投影之前增加有效秩。在CIFAR-10和CIFAR-100上的ViT和Swin实验中，NS-Attn.在所有12个匹配种子的比较中均提高了最终轮次的准确率，平均增益为0.25至0.83个百分点。ViT消融实验显示，一次迭代比两次迭代具有更高的平均准确率。谱分析进一步显示，主导特征值集中度降低，有效秩增加。这些增益带来了额外的推理延迟。
英文摘要
Newton-Schulz (NS) iteration has recently been used in the Muon optimizer to transform update matrices during the training of large language models. Motivated by its spectral effect, we investigate applying NS directly to Transformer attention representations. We introduce Newton-Schulz Attention (NS-Attn.), a parameter-free transformation applied to the output of each attention head. Each head output is arranged as a feature-by-token matrix and normalized by its Frobenius norm. We then apply a finite NS polynomial step and restore the original norm. The objective is to reduce spectral concentration and increase effective rank before standard head merging and output projection. Across ViT and Swin on CIFAR-10 and CIFAR-100, NS-Attn. improves final-epoch accuracy in all 12 matched-seed comparisons, with mean gains of 0.25--0.83 percentage points. ViT ablations show higher mean accuracy with one iteration than with two. Spectral analysis further shows reduced leading-eigenvalue concentration and increased effective rank. These gains incur additional inference latency.
Comments5 pages, 2 figures. Code: https://github.com/039-B/NS-Attention
