过参数化神经网络的临界轨迹
The critical locus of overparameterized neural networks
摘要：
深度学习中损失函数的几何形状的许多方面仍然是神秘的。在本文中，我们努力更好地理解的几何损失函数$L$的过参数化前馈神经网络。在这种情况下，我们确定了几个组件的关键轨迹的$L$和研究他们的几何性质。对于深度为$\ell \geq 4$的网络，我们确定了一个关键点的轨迹，我们称之为星轨迹$S$。在$S$中，我们确定了一个正维子轨迹$C$，它具有以下性质：对于$p \in C$，$p$是一个退化的临界点，并且没有现有的理论结果保证梯度下降不会收敛到$p$。对于非常广泛的网络，我们建立在早期的工作，并表明，所有的临界点的$L$退化，并给出下界的Hessian在每个临界点的零特征值的数量。对于既深又宽的网络，我们比较了我们确定的所有不同临界点家族的Hessian零特征空间的增长率。本文中的结果提供了一个起点，以更定量地了解$L$的临界轨迹的各个组件的属性。
英文摘要：
Many aspects of the geometry of loss functions in deep learning remain mysterious. In this paper, we work toward a better understanding of the geometry of the loss function $L$ of overparameterized feedforward neural networks. In this setting, we identify several components of the critical locus of $L$ and study their geometric properties. For networks of depth $\ell \geq 4$, we identify a locus of critical points we call the star locus $S$. Within $S$ we identify a positive-dimensional sublocus $C$ with the property that for $p \in C$, $p$ is a degenerate critical point, and no existing theoretical result guarantees that gradient descent will not converge to $p$. For very wide networks, we build on earlier work and show that all critical points of $L$ are degenerate, and give lower bounds on the number of zero eigenvalues of the Hessian at each critical point. For networks that are both deep and very wide, we compare the growth rates of the zero eigenspaces of the Hessian at all the different families of critical points that we identify. The results in this paper provide a starting point to a more quantitative understanding of the properties of various components of the critical locus of $L$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
