均匀随机权重如何诱导非均匀偏置：典型插值神经网络借助窄教师网络实现泛化
How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers
AI总结：
本文证明在存在窄教师网络时，参数空间的均匀先验会因网络冗余而诱导函数空间的简单性偏置，使随机插值神经网络以接近教师复杂度的样本复杂度良好泛化。
摘要：
背景。一个主要的理论难题是，为什么过参数化神经网络（NN）在被训练到零损失（即它们对数据进行插值）时仍能良好泛化。通常，神经网络使用随机梯度下降（SGD）或其变体之一进行训练。然而，近期的实证工作考察了一种对数据进行插值的随机神经网络的泛化能力：该神经网络从参数上看似均匀的先验中采样，条件是神经网络能够完美分类训练集。有趣的是，这样采样得到的神经网络通常与经SGD训练的神经网络具有同样好的泛化表现。贡献。我们证明，如果存在一个与标签一致的底层窄“教师NN”，那么这种随机神经网络插值器通常能够良好泛化。具体而言，我们表明，由于神经网络结构中的冗余性，神经网络参数化上的这种“平坦”先验会在神经网络函数上诱导出丰富的先验。特别是，这产生了对更简单函数的偏置，这些函数只需较少的相关参数即可表示——从而使学习所需的样本复杂度近似与教师网络的复杂度（大致即非冗余参数的数量）成正比，而非与学生网络的复杂度成正比。
英文摘要：
Background. A main theoretical puzzle is why over-parameterized Neural Networks (NNs) generalize well when trained to zero loss (i.e., so they interpolate the data). Usually, the NN is trained with Stochastic Gradient Descent (SGD) or one of its variants. However, recent empirical work examined the generalization of a random NN that interpolates the data: the NN was sampled from a seemingly uniform prior over the parameters, conditioned on that the NN perfectly classifies the training set. Interestingly, such a NN sample typically generalized as well as SGD-trained NNs. Contributions. We prove that such a random NN interpolator typically generalizes well if there exists an underlying narrow ``teacher NN'' that agrees with the labels. Specifically, we show that such a `flat' prior over the NN parameterization induces a rich prior over the NN functions, due to the redundancy in the NN structure. In particular, this creates a bias towards simpler functions, which require less relevant parameters to represent -- enabling learning with a sample complexity approximately proportional to the complexity of the teacher (roughly, the number of non-redundant parameters), rather than the student's.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
