\emph{Memorization} in neural networks lacks a precise operational definition and is often inferred from the grokking regime, where training accuracy saturates while test accuracy remains very low. We identify a previously unreported third phase of grokking in this training regime: \emph{anti-grokking}, a late-stage collapse of generalization.
We revisit two canonical grokking setups: a 3-layer MLP trained on a subset of MNIST and a transformer trained on modular addition, but extended training far beyond standard. In both cases, after models transition from pre-grokking to successful generalization, test accuracy collapses back to chance while training accuracy remains perfect, indicating a distinct post-generalization failure mode.
To diagnose anti-grokking, we use the open-source \texttt{WeightWatcher} tool based on HTSR/SETOL theory. The primary signal is the emergence of \emph{Correlation Traps}: anomalously large eigenvalues beyond the Marchenko--Pastur bulk in the empirical spectral density of shuffled weight matrices, which are predicted to impair generalization. As a secondary signal, anti-grokking corresponds to the average HTSR layer quality metric $α$ deviating from $2.0$. Neither metric requires access to the test or training data.
We compare these signals to alternative grokking diagnostic, including $\ell_2$ norms, Activation Sparsity, Absolute Weight Entropy, and Local Circuit Complexity. These track pre-grokking and grokking but fail to identify anti-grokking. Finally, we show that Correlation Traps can induce catastrophic forgetting and/or prototype memorization, and observe similar pathologies in large-scale LLMs, like OSS GPT 20/120B.
Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；University of Washington（华盛顿大学）；University of Michigan（密歇根大学）
Department of Physics, City University of Hong Kong（香港城市大学物理系）；Department of Physics and Astronomy, University of Pennsylvania（宾夕法尼亚大学物理与天文学系）；Innovation Campus Delaware, Air Liquide（Air Liquide创新校园）
Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）；CHU Sainte-Justine Research Center, Montréal（圣朱斯特研究中心，蒙特利尔）；CIFAR AI Chair（CIFAR人工智能席位）
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Arnold-Sommerfeld-Center for Theoretical Physics（阿诺德-索末菲理论物理中心）；Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学）；University of Cambridge（剑桥大学）；Standford University（斯坦福大学）
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China（数学科学国家重点实验室，数学与系统科学研究院，中国科学院，北京100190，中国）；School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China（中国科学院大学数学科学学院，北京100049，中国）；Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China（上海数学与交叉科学研究所（SIMIS），上海，中国）；Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China（浙江省系统健康科学重点实验室，生命科学学院，杭州先进研究院，中国科学院大学，中国科学院，杭州310024，中国）
