困在什么里？重量空间历险记
Stuck in a What? Adventures in Weight Space
摘要：
深度学习研究人员通常认为，收敛模型会陷入局部最小值。最近，一些研究人员观察到，在合理的假设下，绝大多数临界点是鞍点，而不是真正的极小值。这两种描述都表明权重收敛于权重空间中的一个点，无论它是局部最优点还是仅仅是临界点。然而，这两种解释都有可能是不准确的。由于神经网络是典型的过完备网络，很容易通过权值空间以相等的损失来表示大量连续区域的存在。在本文中，我们建立在最近的工作经验表征的误差表面的神经网络。我们通过权重空间分析训练路径，提出证据表明，损失的明显收敛并不对应于权重到达临界点，而是通过权重空间的平坦区域进行大幅度移动。虽然证明神经网络误差表面是全局非凸的是很简单的，但我们证明了误差表面也是局部非凸的，即使在随机初始化打破对称性之后，也是在部分训练之后。
英文摘要：
Deep learning researchers commonly suggest that converged models are stuck in local minima. More recently, some researchers observed that under reasonable assumptions, the vast majority of critical points are saddle points, not true minima. Both descriptions suggest that weights converge around a point in weight space, be it a local optima or merely a critical point. However, it's possible that neither interpretation is accurate. As neural networks are typically over-complete, it's easy to show the existence of vast continuous regions through weight space with equal loss. In this paper, we build on recent work empirically characterizing the error surfaces of neural networks. We analyze training paths through weight space, presenting evidence that apparent convergence of loss does not correspond to weights arriving at critical points, but instead to large movements through flat regions of weight space. While it's trivial to show that neural network error surfaces are globally non-convex, we show that error surfaces are also locally non-convex, even after breaking symmetry with a random initialization and also after partial training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
