发表机构
Arnold-Sommerfeld-Center for Theoretical Physics; Ludwig-Maximilians-Universität München; University of Cambridge; Standford University（阿诺德-索末菲理论物理中心; 慕尼黑路德维希-马克西米利安大学; 剑桥大学; 斯坦福大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出谱理论，揭示权重衰减驱动特征学习导致 grokking 延迟泛化，预测时间尺度由学习率与衰减乘积控制，并在模加法实验中验证。
AI 中文摘要
在 grokking 现象中，对训练数据的早期拟合与后期泛化能力的显著提升之间存在时间间隔。在此延迟期间，训练可以从固定的神经正切核（NTK）机制转变为任务相关的核特征方向持续演化的机制。我们为这种从惰性学习到丰富学习的转变如何产生延迟泛化提供了定量理论。对于使用平方损失和 $L_2$ 权重衰减训练的齐次网络，我们证明在记忆化之后仍存在有限残差，且目标分量中与较小 NTK 特征值相关的残差比例更大。这些残差反馈到 NTK 自身的动力学中，将由此产生的动力学投影到任务相关的谱方向上，可得到一个降阶系统，其中残差驱动的核增长与权重衰减相互竞争。该系统预测：grokking 的时间尺度由学习率与权重衰减的乘积控制；在临界衰减值附近（超过该值，任务对齐的 NTK 结构将无法再支持泛化），特征学习呈对数缓慢；更强的衰减可能完全阻止拟合。我们在模加法任务中测试了这些预测。在一个齐次多层感知机（MLP）中，训练准确率饱和后，任务对齐的傅里叶结构继续在 NTK 中出现，并且一个覆盖不同学习率和权重衰减的 84×90 网络网格恢复了预测的相几何以及泛化时间随学习率和权重衰减的逆乘积缩放。一个单块 Transformer 在 42×45 网格中表现出相似的宏观相结构，并且尽管违反了严格齐次性，仍表现出相同的转变时间缩放。综合这些结果，我们提供了一种机制推导，将拟合后的特征学习与泛化的出现及其在学习率和权重衰减平面中的相结构联系起来。
英文摘要
In grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NTK) regime to one in which task-relevant kernel eigendirections continue to evolve. We provide a quantitative theory for how this transition from lazy to rich learning can produce delayed generalization. For homogeneous networks trained with squared loss and $L_2$ weight decay, we show that a finite residual remains after memorization, with larger residual fractions in target components associated with smaller NTK eigenvalues. These residuals feed back into the dynamics of the NTK itself, and projecting the resulting dynamics onto task-relevant spectral directions yields a reduced system in which residual-driven kernel growth competes with weight decay. This system predicts that the grokking timescale is controlled by the product of learning rate and weight decay, that feature learning slows logarithmically near a critical decay above which task-aligned NTK structure can no longer support generalization, and that stronger decay can prevent fitting altogether. We test these predictions in modular addition. In a homogeneous MLP, task-aligned Fourier structure continues to emerge in the NTK after training accuracy has saturated, and an 84$\times$90-grid of trained networks across varying learning rate and weight decay recovers the predicted phase geometry and inverse-product scaling of the generalization time with learning rate and weight decay. A one-block Transformer shows similar macroscopic phase structure in a 42$\times$45-grid, as well as the same transition-time scaling despite violating exact homogeneity. Together, these results provide a mechanistic derivation connecting post-fit feature learning to both the onset of generalization and its phase structure in the learning rate and weight decay plane.
