Neural networks trained to solve modular arithmetic tasks exhibit grokking, a phenomenon where the test accuracy starts improving long after the model achieves 100% training accuracy in the training process. It is often taken as an example of "emergence", where model ability manifests sharply through a phase transition. In this work, we show that the phenomenon of grokking is not specific to neural networks nor to gradient descent-based optimization. Specifically, we show that this phenomenon occurs when learning modular arithmetic with Recursive Feature Machines (RFM), an iterative algorithm that uses the Average Gradient Outer Product (AGOP) to enable task-specific feature learning with general machine learning models. When used in conjunction with kernel machines, iterating RFM results in a fast transition from random, near zero, test accuracy to perfect test accuracy. This transition cannot be predicted from the training loss, which is identically zero, nor from the test loss, which remains constant in initial iterations. Instead, as we show, the transition is completely determined by feature learning: RFM gradually learns block-circulant features to solve modular arithmetic. Paralleling the results for RFM, we show that neural networks that solve modular arithmetic also learn block-circulant features. Furthermore, we present theoretical evidence that RFM uses such block-circulant features to implement the Fourier Multiplication Algorithm, which prior work posited as the generalizing solution neural networks learn on these tasks. Our results demonstrate that emergence can result purely from learning task-relevant features and is not specific to neural architectures nor gradient descent-based optimization methods. Furthermore, our work provides more evidence for AGOP as a key mechanism for feature learning in neural networks.
Meta AI；Meta Platforms, Inc.（Meta平台公司）；University of Maryland（马里兰大学）
We present a simple neural network that can learn modular arithmetic tasks and exhibits a sudden jump in generalization known as ``grokking''. Concretely, we present (i) fully-connected two-layer networks that exhibit grokking on various modular arithmetic tasks under vanilla gradient descent with the MSE loss function in the absence of any regularization; (ii) evidence that grokking modular arithmetic corresponds to learning specific feature maps whose structure is determined by the task; (iii) analytic expressions for the weights -- and thus for the feature maps -- that solve a large class o…
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
University Paris City & University of Reunion（巴黎城市大学和留尼汪大学）；Dipartimento di Fisica e Chimica Emilio Segrè, Università degli Studi di Palermo（巴勒莫大学埃米利奥·塞格雷物理与化学系）；Thayer School of Engineering, Dartmouth College（达特茅斯学院塞耶工程学院）；EnergyLab, University of Reunion（留尼汪大学能源实验室）；PEACCEL, AI for Biologics（PEACCEL，生物制品人工智能）
A Stochastic--Geometric Theory of Scaling Laws in Grokking
Research Ireland – Centre for Research Training in AI（爱尔兰研究——人工智能研究培训中心）；J.E. Cairnes School of Business & Economics（J.E. 商业与经济学学院）；University of Galway（Galway大学）；Université Laval（拉瓦尔大学）；CIFAR AI Chair（CIFAR人工智能主席）；Mila – Québec AI Institute（魁北克人工智能研究所）；School of Computer Science & Data Science Institute Visual Intelligence Lab（计算机科学与数据科学学院视觉智能实验室）
Department of Physics, City University of Hong Kong（香港城市大学物理系）；Department of Physics and Astronomy, University of Pennsylvania（宾夕法尼亚大学物理与天文学系）；Innovation Campus Delaware, Air Liquide（Air Liquide创新校园）
Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；University of Washington（华盛顿大学）；University of Michigan（密歇根大学）
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）；Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS)（上海数学与交叉学科研究院）；Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（浙江省系统健康科学重点实验室，中国科学院大学杭州高等研究院生命科学学院）
