arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.13011cs.LGphysics.bio-phq-bio.BM
收藏

关于深度学习中的幂律Hessian谱

On the Power-Law Hessian Spectrums in Deep Learning

Baidu Research(百度研究院);RIKEN Center for Brain Science(理化学研究所脑科学中心)

Zeke Xie, Qian-Yuan Tang, Yunfeng Cai, Mingming Sun, Ping Li

摘要:

众所周知,深度损失景观的Hessian关系到深度学习的优化、泛化甚至鲁棒性。最近的工作经验性地发现,深度学习中的Hessian谱具有由少量大特征值和大量近零特征值组成的双分量结构。然而,Hessian谱背后的理论机制或数学仍然在很大程度上未被探索。据我们所知,我们是第一个证明训练有素的深度神经网络的Hessian谱表现出简单的幂律结构的人。受统计物理理论和天然蛋白质光谱分析的启发,我们提供了一个最大熵理论解释为什么幂律结构存在,并提出了蛋白质进化和深层神经网络训练之间的光谱平行。通过进行广泛的实验,我们进一步使用幂律谱框架作为一个有用的工具来探索深度学习的多种新行为。

英文摘要:

It is well-known that the Hessian of deep loss landscape matters to optimization, generalization, and even robustness of deep learning. Recent works empirically discovered that the Hessian spectrum in deep learning has a two-component structure that consists of a small number of large eigenvalues and a large number of nearly-zero eigenvalues. However, the theoretical mechanism or the mathematical behind the Hessian spectrum is still largely under-explored. To the best of our knowledge, we are the first to demonstrate that the Hessian spectrums of well-trained deep neural networks exhibit simple power-law structures. Inspired by the statistical physical theories and the spectral analysis of natural proteins, we provide a maximum-entropy theoretical interpretation for explaining why the power-law structure exist and suggest a spectral parallel between protein evolution and training of deep neural networks. By conducing extensive experiments, we further use the power-law spectral framework as a useful tool to explore multiple novel behaviors of deep learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑