Supervised deep learning involves the training of neural networks with a large number $N$ of parameters. For large enough $N$, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as $N$ grows past a certain threshold $N^{*}$. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with $N$. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations $\|f_{N}-\bar{f}_{N}\|\sim N^{-1/4}$ of the neural net output function $f_{N}$ around its expectation $\bar{f}_{N}$. These affect the generalization error $ε_{N}$ for classification: under natural assumptions, it decays to a plateau value $ε_{\infty}$ in a power-law fashion $\sim N^{-1/2}$. This description breaks down at a so-called jamming transition $N=N^{*}$. At this threshold, we argue that $\|f_{N}\|$ diverges. This result leads to a plausible explanation for the cusp in test error known to occur at $N^{*}$. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond $N^{*}$, and averaging their outputs.
A Unified Generalization Framework for Model Merging: Trade-offs, Non-Linearity, and Scaling Laws
National University of Defense Technology(国防科技大学);Wuhan University(武汉大学);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学信息科学与技术学院深圳校区);Hebei University of Technology(河北工业大学)
Does Compression Imply Generalization? A Minimum Description Length Perspective
Leiden University(莱顿大学);Centrum Wiskunde & Informatica (CWI)(数学和计算机科学中心);Hefei University of Technology(合肥工业大学);Great Bay University(大湾区大学);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);UiT The Arctic University of Norway(挪威北极大学)
Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
John A. Paulson School of Engineering And Applied Sciences, Harvard University(哈佛大学约翰·A·保罗森工程与应用科学学院);Center for Brain Science, Harvard University(哈佛大学脑科学中心);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学自然与人工智能研究学院)
Effective Hamiltonian description on monitored Majorana chains: correlated power-law hoppings and unconventional entanglement scaling
University of Tokyo(东京大学);RIKEN Pioneering Research Institute (PRI)(理化学研究所先锋研究本部);University of Warwick(华威大学);RIKEN Center for Interdisciplinary Theoretical and Mathematical Sciences (iTHEMS)(理化学研究所交叉理论与数学科学中心);RIKEN(理化学研究所);Ritsumeikan University(立命馆大学)