发表机构
Korea Institute for Advanced Study; Center for AI and Natural Sciences（韩国高等研究院; 人工智能与自然科学中心）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对延迟泛化问题，在线性模型及非线性神经网络中确定了可精确求解的后期松弛机制，揭示grokking子空间并推导相关预测，验证理论恒等式，还在模块化加法中观察到延迟泛化，展现理论与实际的契合。
AI 中文摘要
尽管进行了大量实证研究，但延迟泛化（即grokking）仍未被充分理解。我们确定了一种在线性模型中可精确求解的后期松弛机制，该模型通过全批量重球优化和权重衰减进行训练，并对非线性神经网络进行了局部二次扩展。我们的分析揭示了经验零空间中一个独特的群体活跃成分，即grokking子空间。沿着这个子空间，训练预测保持不变，权重衰减成为唯一的恢复力，导致由精确的离散时间和连续时间定律控制的缓慢耗散松弛。我们表明只有这个子空间对群体风险的缓慢渐近衰减有贡献，并推导出grokking时间的明确迭代尺度预测，在弱正则化 regime 中恢复了熟悉的$(1-\beta)/(\eta\lambda)$缩放。该理论进一步预测了优化器选择的不同影响，区分了耦合的$L_2$正则化和解耦的权重衰减，并对修改grokking成分的干预产生因果预测。我们在一个合成模型中验证了所有无拟合参数的理论恒等式，其中每个子空间和松弛率都可以以封闭形式计算。我们还在模块化加法中观察到了真正的延迟泛化，其中测量到的延迟遵循预测的缩放，后期松弛与理论时钟密切一致。
英文摘要
Delayed generalization, or grokking, remains poorly understood despite extensive empirical study. We identify an exactly solvable late-time relaxation mechanism for grokking in linear models trained with full-batch heavy-ball optimization and weight decay, together with a locally quadratic extension to nonlinear neural networks. Our analysis reveals a distinguished population-active component of the empirical null space, which we call the grokking subspace. Along this subspace, the training predictions remain unchanged, leaving weight decay as the sole restoring force and giving rise to a slow dissipative relaxation governed by an exact discrete-time and continuous-time law. We show that only this subspace contributes to the slow asymptotic decay of the population risk and derive explicit iteration-scale predictions for the grokking time, recovering the familiar $(1-β)/(ηλ)$ scaling in the weak-regularization regime. The theory further predicts distinct effects of optimizer choice, distinguishing coupled $L_2$ regularization from decoupled weight decay, and yields causal predictions for interventions that modify the grokking component. We verify all theoretical identities without fitted parameters in a synthetic model where every subspace and relaxation rate is computable in closed form. We further observe genuine delayed generalization in modular addition, where the measured delay follows the predicted scaling and the late-time relaxation agrees closely with the theoretical clock.
Comments33 pages, 5 figures
