从参数化不足到过度参数化的干扰过渡影响损失景观和泛化
A jamming transition from under- to over-parametrization affects loss landscape and generalization
摘要:
我们认为,在完全连接的网络中的相变划定的过度和欠参数化制度,拟合可以或不能实现。在某些一般条件下,我们表明,这种转变是尖锐的铰链损失。在整个过参数化的制度中,在训练期间不会遇到损失的最小值,因为要满足的约束的数量太小而不能妨碍最小化。我们的研究结果支持这种转变与网络的泛化属性之间的联系:当我们从一个欠参数化的网络开始增加给定模型的参数数量时,我们观察到泛化误差显示出三个阶段:(i)初始衰变,(ii)增加,直到过渡点-在那里它显示一个尖点-和(iii)缓慢衰减到一个常数的其余部分的过度参数化制度。因此,我们确定了经典的过度拟合现象发生的区域,以及模型不断改进的区域,与以前对现代神经网络的经验观察一致。
英文摘要:
We argue that in fully-connected networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. Under some general conditions, we show that this transition is sharp for the hinge loss. In the whole over-parametrized regime, poor minima of the loss are not encountered during training since the number of constraints to satisfy is too small to hamper minimization. Our findings support a link between this transition and the generalization properties of the network: as we increase the number of parameters of a given model, starting from an under-parametrized network, we observe that the generalization error displays three phases: (i) initial decay, (ii) increase until the transition point --- where it displays a cusp --- and (iii) slow decay toward a constant for the rest of the over-parametrized regime. Thereby we identify the region where the classical phenomenon of over-fitting takes place, and the region where the model keeps improving, in line with previous empirical observations for modern neural networks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。