发表机构
Delft University of Technology; HDL TypeTech; IBM Infrastructure（代尔夫特理工大学; HDL TypeTech; IBM基础设施部门）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出基于模式连通性和低损失区域几何的分析框架，发现训练与验证划分导致的低损失区域错位是顿悟现象产生的根源，并通过反例挑战现有解释。
AI 中文摘要
顿悟（Grokking）指的是模型在已经过拟合训练集之后，验证集泛化能力延迟出现的现象。尽管这一现象最初是在使用Transformer训练的小型算法任务中观察到的，但其背后的机制至今仍未明确。在本工作中，我们基于模式连通性和低损失区域的几何结构，开发了一个分析框架。该框架预测，标准的模算术设置并不总是产生顿悟现象：在保持对称性的训练/验证划分下，我们观察到一个稳定的反顿悟（anti-grokking）案例，其中验证性能无法恢复。这一反例对现有关于顿悟的若干相关性解释提出了挑战。更广泛地，我们的分析框架和结果进一步表明，当训练和验证划分所诱导的低损失区域错位时，顿悟现象就会出现。一旦这些区域变得良好对齐，仅靠训练超参数无法产生顿悟，观察到的动态将坍缩为可训练或不可训练的行为。
英文摘要
Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism remains unsettled. In this work, we develop an analysis framework based on mode connectivity and the geometry of low-loss regions. The framework predicts that the standard modular-arithmetic setting does not always produce grokking: under a symmetry-preserving train/validation split, we observe a stable anti-grokking case in which validation performance does not recover. This counterexample challenges several existing correlational explanations of grokking. More broadly, our analysis framework and results further suggest that grokking arises when the low-loss regions induced by the training and validation partitions are misaligned. Once these regions become well aligned, training hyperparameters alone cannot produce grokking and the observed dynamics collapse to either trainable or non-trainable behavior.
Comments23 pages, 23 figures
