非对称谷：超越局部极值的尖与平
Asymmetric Valleys: Beyond Sharp and Flat Local Minima
摘要：
尽管其损失函数具有非凸性质，但已知深度神经网络在使用随机梯度下降（SGD）优化时具有良好的泛化能力。最近的工作推测，具有适当配置的SGD能够找到宽且平坦的局部最小值，这被认为与良好的泛化性能相关。在本文中，我们观察到现代深度网络的局部极小值不仅仅是平坦或尖锐的。具体来说，在局部最小值处存在许多不对称方向，使得损耗沿着一侧突然增加，并且沿着相对侧缓慢增加-我们正式地将这种最小值定义为不对称谷。在温和的假设下，我们证明了，对于不对称的山谷，偏向于平坦的一面的解决方案推广比精确的极小。此外，我们表明，简单地平均沿着SGD轨迹的权重会产生这样的偏见的解决方案隐含。这为Izmailov等人（2018）观察到的有趣现象提供了理论解释。此外，我们的经验发现，批量归一化（BN）似乎是一个主要原因不对称的山谷。
英文摘要：
Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good generalization performance. In this paper, we observe that local minima of modern deep networks are more than being flat or sharp. Specifically, at a local minimum there exist many asymmetric directions such that the loss increases abruptly along one side, and slowly along the opposite side--we formally define such minima as asymmetric valleys. Under mild assumptions, we prove that for asymmetric valleys, a solution biased towards the flat side generalizes better than the exact minimizer. Further, we show that simply averaging the weights along the SGD trajectory gives rise to such biased solutions implicitly. This provides a theoretical explanation for the intriguing phenomenon observed by Izmailov et al. (2018). In addition, we empirically find that batch normalization (BN) appears to be a major cause for asymmetric valleys.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
