Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
The Abdus Salam International Centre for Theoretical Physics (ICTP)(阿布杜斯·萨拉姆国际理论物理中心);International School for Advanced Studies (SISSA)(国际先进研究学院)
Jean Barbier, Francesco Camilli, Minh-Toan Nguyen, Mauro Pastore, Rudy Skerk
备注:
This submission has been superseded by arxiv:2510.24616, which describes a unified theory to derive the phase diagram of the model, accounts for non-homogeneous readouts, pushes forward the approach to deeper targets/models, and has been published in Phys. Rev. X 16, 031014 (2026)
We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width $k$ and input dimension $d$ are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size $n$ scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters $kd+k$ and of data $n$ are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate $n/d^2$, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms.
Univ Rennes(雷恩大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);IRISA(信息与信号处理研究所);University College London(伦敦大学学院);Université de Lille(里尔大学);Ecole Normale Supérieure(高等师范学院);PSL Research University(巴黎文理研究大学)
ASNI: Adaptive Structured Noise Injection for shallow and deep neural networks
MINES ParisTech, PSL Research University(巴黎高等矿业学院,PSL研究大学);Institut Curie, PSL Research University(居里研究所,PSL研究大学);INSERM(法国国家健康与医学研究院);Google Brain(谷歌大脑)
The Gaussian equivalence of generative models for learning with shallow neural networks
International School of Advanced Studies (SISSA)(国际高等研究学院(SISSA));Duke University(杜克大学);Université PSL(巴黎文理研究大学);Sorbonne Université(索邦大学);Université Paris-Diderot(巴黎第七大学)
A Basin-Selection Perspective on Grokking via Singular Learning Theory
Department of Computer Science(计算机科学系);Department of Mathematics(数学系);Mathematical Institute(数学研究所);University of Pisa(比萨大学);Imperial College London(伦敦帝国学院);University of Oxford(牛津大学);Section of Mathematics and Artificial Intelligence(数学与人工智能系);Max Planck Institute of Molecular Cell Biology and Genetics(马克斯·普朗克分子细胞生物学和遗传学研究所);Center for Systems Biology(系统生物学中心);Faculty of Mathematics(数学系);TU Dresden(德累斯顿技术大学)