arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.18530stat.MLcond-mat.dis-nncond-mat.stat-mechcs.ITcs.LGmath.IT
收藏

在插值附近的宽浅层神经网络中的最优泛化与学习转变

Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation

The Abdus Salam International Centre for Theoretical Physics (ICTP)(阿布杜斯·萨拉姆国际理论物理中心);International School for Advanced Studies (SISSA)(国际先进研究学院)

Jean Barbier, Francesco Camilli, Minh-Toan Nguyen, Mauro Pastore, Rudy Skerk

AI总结:

研究宽浅层神经网络在插值附近的情况,针对样本大小与输入维度二次缩放时,给出近似贝叶斯最优泛化误差的有效理论,分析一般权重分布,发现相变并揭示插值附近有高预测解但难被实际算法找到。

摘要:

我们考虑一个监督学习的师生模型,其中有一个经过充分训练的两层神经网络,其宽度$k$和输入维度$d$很大且成比例。对于样本大小$n$与输入维度二次缩放的情况,即围绕可训练参数数量$kd + k$和数据数量$n$相当的插值阈值,我们为任何激活函数提供了一种有效理论来近似网络的贝叶斯最优泛化误差。我们的分析处理一般的权重分布。我们发现了一个将“通用”阶段与“专业化”阶段分开的不连续相变。在第一个阶段,泛化误差与权重分布无关,并随采样率$n/d^2$缓慢衰减,学生仅学习教师权重的一些非线性组合。在第二个阶段,误差与权重分布有关,并且由于学生向教师网络的对齐而衰减得更快。因此,我们揭示了在插值附近存在一个高度预测性的解,然而实际算法可能很难找到它。

英文摘要:

We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width $k$ and input dimension $d$ are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size $n$ scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters $kd+k$ and of data $n$ are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate $n/d^2$, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑