Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that grokking is a norm-driven representational phase transition in regularised training dynamics, and establish the Norm-Separation Delay Law: $T_{\mathrm{grok}} - T_{\mathrm{mem}} = Θ(γ_{\mathrm{eff}}^{-1} \log(\|θ_{\mathrm{mem}}\|^2 / \|θ_{\mathrm{post}}\|^2))$, where $γ_{\mathrm{eff}}$ is the optimiser's effective contraction rate ($γ_{\mathrm{eff}} = ηλ$ for SGD, $γ_{\mathrm{eff}} \ge ηλ$ for AdamW). The upper bound follows from a discrete Lyapunov contraction argument; the matching lower bound from dynamical constraints of regularised first-order optimisation. Across 293 training runs spanning modular addition, modular multiplication, and sparse parity, we confirm three falsifiable predictions: inverse scaling with weight decay ($R^2 = 0.97$), inverse scaling with learning rate ($R^2 = 残差 0.92$), and logarithmic dependence on the norm ratio (Pearson $r = 0.91$). A fourth finding reveals that grokking requires an optimiser capable of decoupling memorisation from contraction: SGD fails entirely at the same hyperparameters where AdamW reliably groks. These results reframe grokking not as a mysterious optimisation artefact but as a predictable consequence of norm separation between competing interpolating representations. We further derive a practical three-input algorithm that predicts grokking delay at memorisation time with 34.6% mean absolute error (bootstrap 95% CI [30.0%, 39.4%], $N=60$ seeds), enabling principled early stopping.
英文摘要：
Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that grokking is a norm-driven representational phase transition in regularised training dynamics, and establish the Norm-Separation Delay Law: $T_{\mathrm{grok}} - T_{\mathrm{mem}} = Θ(γ_{\mathrm{eff}}^{-1} \log(\|θ_{\mathrm{mem}}\|^2 / \|θ_{\mathrm{post}}\|^2))$, where $γ_{\mathrm{eff}}$ is the optimiser's effective contraction rate ($γ_{\mathrm{eff}} = ηλ$ for SGD, $γ_{\mathrm{eff}} \ge ηλ$ for AdamW). The upper bound follows from a discrete Lyapunov contraction argument; the matching lower bound from dynamical constraints of regularised first-order optimisation. Across 293 training runs spanning modular addition, modular multiplication, and sparse parity, we confirm three falsifiable predictions: inverse scaling with weight decay ($R^2 = 0.97$), inverse scaling with learning rate ($R^2 = 0.92$), and logarithmic dependence on the norm ratio (Pearson $r = 0.91$). A fourth finding reveals that grokking requires an optimiser capable of decoupling memorisation from contraction: SGD fails entirely at the same hyperparameters where AdamW reliably groks. These results reframe grokking not as a mysterious optimisation artefact but as a predictable consequence of norm separation between competing interpolating representations. We further derive a practical three-input algorithm that predicts grokking delay at memorisation time with 34.6% mean absolute error (bootstrap 95% CI [30.0%, 39.4%], $N=60$ seeds), enabling principled early stopping.
Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS Center)（三机构神经影像与数据科学转化研究中心）；Beijing Institute of Technology（北京理工大学）；Georgia Institute of Technology（佐治亚理工学院）
GrokAlign: Geometric Characterisation and Acceleration of Grokking
Department of Electrical and Computer Engineering, Rice University（电气与计算机工程系， Rice大学）；Google Research（谷歌研究院）；Department of Computer Science, Brown University（计算机科学系， Brown大学）
Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent
Center for Computational Neuroscience（计算神经科学中心）；Flatiron Institute（Flatiron研究所）；Department of Physics（物理系）；Harvard University（哈佛大学）；Kempner Institute（Kempner研究所）；New York University（纽约大学）
Grokking Beyond the Euclidean Norm of Model Parameters
Mila, Quebec AI Institute, Montréal, Quebec, Canada（魁北克AI研究所）；CHU Sainte-Justine Research Center, Montréal, Quebec, Canada（圣-朱斯特研究中心）；CIFAR AI Chair（CIFAR人工智能席位）
A Stochastic--Geometric Theory of Scaling Laws in Grokking
Research Ireland – Centre for Research Training in AI（爱尔兰研究——人工智能研究培训中心）；J.E. Cairnes School of Business & Economics（J.E. 商业与经济学学院）；University of Galway（Galway大学）；Université Laval（拉瓦尔大学）；CIFAR AI Chair（CIFAR人工智能主席）；Mila – Québec AI Institute（魁北克人工智能研究所）；School of Computer Science & Data Science Institute Visual Intelligence Lab（计算机科学与数据科学学院视觉智能实验室）
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Arnold-Sommerfeld-Center for Theoretical Physics（阿诺德-索末菲理论物理中心）；Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学）；University of Cambridge（剑桥大学）；Standford University（斯坦福大学）
To Grok Grokking: Provable Grokking in Ridge Regression
Department of Computer Science, Purdue University, West Lafayette, IN, USA（普渡大学计算机科学系）；Department of Computer Science（计算机科学系）；Applied Mathematics, Weizmann Institute of Science, Israel（科学应用数学系，魏茨曼研究院）；Stein Faculty of Computer（斯坦因计算机科学学院）；Information Science, Ben-Gurion University of the Negev, Israel（信息科学系，本· Gurion 军事大学）
Raymond and Beverly Sackler School of Physics and Astronomy, Tel Aviv University, Tel Aviv 69978, Israel（雷蒙德与贝弗利·萨克勒物理与天文学学院，特拉维夫大学）；École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland（洛桑联邦理工学院（EPFL），洛桑，瑞士）
Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking
Ting Han Lamarr Institute, TU Dortmund, Germany（廷汉拉马尔研究所，杜伊斯堡-艾森大学，德国）；Institute for AI in Medicine, UK Essen（人工智能医学研究所，英国埃森）；Research Center Trustworthy Data Science and Security of the University Alliance Ruhr, TU Dortmund, Germany（trustworthy数据科学与安全研究所以及鲁尔大学联盟，杜伊斯堡-艾森大学，德国）；Ruhr University Bochum, Germany（波恩鲁尔大学，德国）；Department of Physics, TU Dortmund, Germany（物理系，杜伊斯堡-艾森大学，德国）；Lamarr Institute, TU Dortmund, Germany（廷汉拉马尔研究所，杜伊斯堡-艾森大学，德国）
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
University Paris City & University of Reunion（巴黎城市大学和留尼汪大学）；Dipartimento di Fisica e Chimica Emilio Segrè, Università degli Studi di Palermo（巴勒莫大学埃米利奥·塞格雷物理与化学系）；Thayer School of Engineering, Dartmouth College（达特茅斯学院塞耶工程学院）；EnergyLab, University of Reunion（留尼汪大学能源实验室）；PEACCEL, AI for Biologics（PEACCEL，生物制品人工智能）
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
The article is devoted to investigating the application of hedging strategies to online expert weight allocation under delayed feedback. As the main result, we develop the General Hedging algorithm $\mathcal{G}$ based on the exponential reweighing of experts' losses. We build the artificial probabilistic framework and use it to prove the adversarial loss bounds for the algorithm $\mathcal{G}$ in the delayed feedback setting. The designed algorithm $\mathcal{G}$ can be applied to both countable and continuous sets of experts. We also show how algorithm $\mathcal{G}$ extends classical Hedge (Mul…
MIT（麻省理工学院）；Institute for AI and Fundamental Interactions（人工智能与基础交互研究所）
We aim to understand grokking, a phenomenon where models generalize long after overfitting their training set. We present both a microscopic analysis anchored by an effective theory and a macroscopic analysis of phase diagrams describing learning performance across hyperparameters. We find that generalization originates from structured representations whose training dynamics and dependence on training set size can be predicted by our effective theory in a toy setting. We observe empirically the presence of four learning phases: comprehension, grokking, memorization, and confusion. We find repr…
Online learning with delayed feedback has received increasing attention recently due to its several applications in distributed, web-based learning problems. In this paper we provide a systematic study of the topic, and analyze the effect of delay on the regret of online learning algorithms. Somewhat surprisingly, it turns out that delay increases the regret in a multiplicative way in adversarial problems, and in an additive way in stochastic problems. We give meta-algorithms that transform, in a black-box fashion, algorithms developed for the non-delayed case into ones that can handle the pre…
