尺度不变Transformer中Grokking的低秩衰减：谱几何视角
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
AI总结：
针对尺度不变Transformer中权重衰减无法简化归一化层函数的问题，提出低秩衰减（LRD）正则化器，通过核范数子梯度的切向分量压缩奇异值，在模算术任务中加速有效秩下降并扩展延迟泛化（grokking）的数据边界。
摘要：
现代Transformer架构经常采用归一化机制，如RMSNorm和Query-Key归一化，使得模型的部分相对于权重幅度近似尺度不变。在这种机制下，标准的Frobenius范数权重衰减仅沿权重空间的径向方向作用，无法直接简化归一化层所表示的函数。我们通过这一视角研究小规模算法任务中的grokking现象，并提出\emph{低秩衰减}（LRD），一种类似核范数的谱正则化器，其子梯度——极因子$UV^\top$——即使在尺度不变设置中也保留切向分量。这一区别具有具体的动力学后果：在模型记忆训练集且任务梯度消失后，L2衰减无法再重塑权重谱，而LRD则以类似$\ell_1$的方式继续压缩奇异值。在模算术任务中，我们发现LRD诱导Query/Key矩阵的快速有效秩下降，并扩展了延迟泛化（grokking）发生的数据分数边界。我们进一步通过核范数子微分在低秩流形附近的“针到扇”展开，提供了谱几何解释。
英文摘要：
Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes. In this regime, standard Frobenius-norm weight decay acts purely along the radial direction of the weight space and cannot directly simplify the function represented by the normalized layer. We study grokking in small algorithmic tasks through this lens and propose \emph{Low-Rank Decay} (LRD), a nuclear-norm-like spectral regularizer whose subgradient -- the polar factor $UV^\top$ -- retains a tangential component even in the scale-invariant setting. This distinction has a concrete dynamical consequence: after the model memorizes the training set and task gradients vanish, L2 decay can no longer reshape the weight spectrum, whereas LRD continues to compress singular values in an $\ell_1$-like fashion. On modular arithmetic tasks, we find that LRD induces rapid effective-rank collapse in Query/Key matrices and expands the data-fraction boundary at which delayed generalization (grokking) occurs. We further provide a spectral-geometric interpretation through the ``needle-to-fan'' expansion of the nuclear-norm subdifferential near low-rank strata.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
