We explore the critical data size in language models, a threshold that marks a fundamental shift from quick memorization to slow generalization. We formalize the phase transition under the grokking configuration into the Data Efficiency Hypothesis and identify data insufficiency, sufficiency, and surplus regimes in language models training dynamics. We develop a grokking configuration to reproduce grokking on simplistic language models stably by rescaling initialization and weight decay. We show that generalization occurs only when language models reach a critical size. We analyze grokking across sample-wise and model-wise, verifying the proposed data efficiency hypothesis. Our experiments reveal smoother phase transitions occurring at the critical dataset size for language datasets. As the model size increases, this critical point also becomes larger, indicating that larger models require more data. Our results deepen the understanding of language model training, offering a novel perspective on the role of data in the learning mechanism of language models.
A Basin-Selection Perspective on Grokking via Singular Learning Theory
Department of Computer Science(计算机科学系);Department of Mathematics(数学系);Mathematical Institute(数学研究所);University of Pisa(比萨大学);Imperial College London(伦敦帝国学院);University of Oxford(牛津大学);Section of Mathematics and Artificial Intelligence(数学与人工智能系);Max Planck Institute of Molecular Cell Biology and Genetics(马克斯·普朗克分子细胞生物学和遗传学研究所);Center for Systems Biology(系统生物学中心);Faculty of Mathematics(数学系);TU Dresden(德累斯顿技术大学)
Grokking phase transitions in learning local rules with gradient descent
Faculty of computer and information science, University of Ljubljana(卢布尔雅那大学计算机与信息科学学院);Faculty of mathematics and physics, University of Ljubljana(卢布尔雅那大学数学与物理学院)
Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS Center)(三机构神经影像与数据科学转化研究中心);Beijing Institute of Technology(北京理工大学);Georgia Institute of Technology(佐治亚理工学院)
Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Raymond and Beverly Sackler School of Physics and Astronomy, Tel Aviv University, Tel Aviv 69978, Israel(雷蒙德与贝弗利·萨克勒物理与天文学学院,特拉维夫大学);École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland(洛桑联邦理工学院(EPFL),洛桑,瑞士)
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Arnold-Sommerfeld-Center for Theoretical Physics(阿诺德-索末菲理论物理中心);Ludwig-Maximilians-Universität München(慕尼黑路德维希-马克西米利安大学);University of Cambridge(剑桥大学);Standford University(斯坦福大学)
School of Science, Huzhou University, Huzhou 313000, China(湖州大学科学学院);Mandelstam Institute for Theoretical Physics, School of Physics, University of the Witwatersrand, Private Bag 3, Wits 2050, South Africa(威特沃特斯兰大学物理学院)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Machine Learning Platform Department, Tencent(腾讯机器学习平台部);School of Computer Science, Wuhan University(武汉大学计算机学院)
Raymond and Beverly Sackler School of Physics and Astronomy, Tel-Aviv University, Tel-Aviv 69978, Israel(雷蒙德与贝弗利·萨克勒物理与天文学学院,特拉维夫大学);École Polytechnique Fédérale de Lausanne (EPFL), Switzerland(洛桑联邦理工学院(EPFL),瑞士);Racah Institute of Physics, The Hebrew University of Jerusalem, Edmond J. Safra Campus, Jerusalem 9190401, Israel(拉卡姆物理研究所,耶路撒冷希伯来大学);Selim Benin School of Computer Science and Engineering, The Hebrew University of Jerusalem, Edmond J. Safra Campus, Jerusalem 9190401, Israel(塞利姆·本因计算机科学与工程学院,耶路撒冷希伯来大学)
Evaluating Large Language Model Creativity from a Literary Perspective
Imperial College London(帝国理工学院);Google DeepMind(谷歌DeepMind);Institute of Historical Research, School of Advanced Study, University of London(伦敦大学高级研究院历史研究所);University of Southampton(南安普顿大学)
CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective
School of Computer Science, University of Science and Technology of China(科学技术大学计算机科学学院);School of Biomedical Engineering, USTC(USTC生物医学工程学院);Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research(苏州先进研究院数据黑暗实验室、奇迹中心);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院)