arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1901.01608cond-mat.dis-nncs.LG
收藏

深度学习中多参数泛化的尺度描述

Scaling description of generalization with number of parameters in deep learning

École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院);École Normale Supérieure, PSL Research University(巴黎高等师范学院,PSL研究大学)

Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, Matthieu Wyart

摘要:

监督式深度学习涉及使用大量$N$参数训练神经网络。对于足够大的$N$,在所谓的过参数化状态下,基本上可以拟合训练数据点。基于稀疏性的论证表明,当$N$增长超过某个阈值$N^{*}$时,泛化误差会增加。相反,实证研究表明,在过参数化的制度,泛化误差不断减少与$N$。我们通过一个新的框架来解决这个矛盾。我们依赖于所谓的神经切线内核,它连接大型神经网络的内核方法,表明初始化导致有限大小的随机波动$\|f_{N}-\bar{f}_{N}\|\sim N^{-1/4}$的神经网络输出函数$f_{N}$周围的期望值$\bar{f}_{N}$。这些影响分类的泛化误差$ε_{N}$:在自然假设下,它以幂律方式$\sim N^{-1/2}$衰减到平台值$ε_{\infty}$。该描述在所谓的干扰转变$N=N^{*}$处中断。在这个阈值,我们认为,$\|f_{N}\|$分歧。该结果为已知在$N^{*}$发生的测试误差中的尖点提供了合理的解释。我们的研究结果证实了广泛的经验观察MNIST和CIFAR图像数据集。我们的分析最后表明,给定一个计算包络,最小的泛化误差是使用几个中等大小的网络,刚刚超过$N^{*}$,并平均其输出。

英文摘要:

Supervised deep learning involves the training of neural networks with a large number $N$ of parameters. For large enough $N$, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as $N$ grows past a certain threshold $N^{*}$. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with $N$. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations $\|f_{N}-\bar{f}_{N}\|\sim N^{-1/4}$ of the neural net output function $f_{N}$ around its expectation $\bar{f}_{N}$. These affect the generalization error $ε_{N}$ for classification: under natural assumptions, it decays to a plateau value $ε_{\infty}$ in a power-law fashion $\sim N^{-1/2}$. This description breaks down at a so-called jamming transition $N=N^{*}$. At this threshold, we argue that $\|f_{N}\|$ diverges. This result leads to a plausible explanation for the cusp in test error known to occur at $N^{*}$. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond $N^{*}$, and averaging their outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑