It is well-known that the Hessian of deep loss landscape matters to optimization, generalization, and even robustness of deep learning. Recent works empirically discovered that the Hessian spectrum in deep learning has a two-component structure that consists of a small number of large eigenvalues and a large number of nearly-zero eigenvalues. However, the theoretical mechanism or the mathematical behind the Hessian spectrum is still largely under-explored. To the best of our knowledge, we are the first to demonstrate that the Hessian spectrums of well-trained deep neural networks exhibit simple power-law structures. Inspired by the statistical physical theories and the spectral analysis of natural proteins, we provide a maximum-entropy theoretical interpretation for explaining why the power-law structure exist and suggest a spectral parallel between protein evolution and training of deep neural networks. By conducing extensive experiments, we further use the power-law spectral framework as a useful tool to explore multiple novel behaviors of deep learning.
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
NYU(纽约大学);Facebook(脸书);Facebook AI Research(脸书人工智能研究院);Institut de Physique Théorique(理论物理研究所);Université Paris Saclay(巴黎萨克雷大学);CEA(法国原子能和替代能源委员会)
通过Hessian矩阵研究了常见损失曲面的性质。特别是,在深度学习的背景下,我们根据经验表明,Hessian的频谱由两部分组成:(1)以零为中心的块,(2)远离块的离群值。我们为Sagun et al.(2016)提出的以下假设提供了数值证据和数学证明:固定数据,增加参数的数量只是扩展了频谱的大部分;固定维度和改变数据(例如添加更多的聚类或使数据更难分离)只影响离群值。我们相信,我们的观察有显着的影响,在高维非凸优化。首先,这种景观的平坦性(可以用黑森的奇点来衡量)意味着吸引盆地的经典概念可能是相当误导的。宽/窄流域的讨论可能需要一个新的视角,围绕过度参数化和冗余,能够在景观的底部创建大型连接组件。其次,少数大特征值对数据分布的依赖性可以与模型输出梯度的协方差矩阵的谱相关联。考虑到这一点,我们可以重新评估模型的数据架构算法框架内的连接,希望它能在现代应用中揭示高维和非凸空间的几何。特别是,我们提出了一种情况下,连接两个观察:小批量和大批量的梯度下降似乎收敛到不同的盆地的吸引力,但我们表明,他们实际上是通过其平坦的区域连接,因此属于同一个盆地。
On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature
School of Physics, Peking University(北京大学物理系);Peking University Chengdu Academy for Advanced Interdisciplinary Biotechnologies(北京大学成都先进交叉生物技术研究院);Flatiron Institute(Flatiron研究所)
On progressive sharpening, flat minima and generalisation
Mathematical Institute for Data Science(数据科学数学研究所);Johns Hopkins University(约翰斯·霍普金斯大学);Australian Institute for Machine Learning(澳大利亚机器学习研究所);University of Adelaide(阿德莱德大学)