arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.06740cs.LGmath-phmath.MPstat.ML
收藏

随机矩阵理论在深度学习中的出现

Appearance of Random Matrix Theory in Deep Learning

University of Bristol(布里斯托大学);University of Oxford(牛津大学)

Nicholas P Baskerville, Diego Granziol, Jonathan P Keating

摘要:

我们研究了人工神经网络的损失表面Hessian的局部谱统计,在那里我们发现了在几个网络架构和数据集上与高斯正交包络统计的极好的一致性。这些结果揭示了随机矩阵理论对神经网络建模的适用性,并表明它在深度学习中的损失表面研究中以前未被认识到的作用。受这些观察的启发,我们提出了一个新的模型的真实损失表面的神经网络,与我们的观察,它允许海森谱密度与秩退化和离群值,广泛观察到在实践中,并预测损失梯度的权重空间的距离的函数越来越独立。我们进一步研究了真实损失面在神经网络中的重要性,并发现,与以前的工作相比,定位全局最小值的指数硬度对实现最先进的性能具有实际影响。

英文摘要:

We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover excellent agreement with Gaussian Orthogonal Ensemble statistics across several network architectures and datasets. These results shed new light on the applicability of Random Matrix Theory to modelling neural networks and suggest a previously unrecognised role for it in the study of loss surfaces in deep learning. Inspired by these observations, we propose a novel model for the true loss surfaces of neural networks, consistent with our observations, which allows for Hessian spectral densities with rank degeneracy and outliers, extensively observed in practice, and predicts a growing independence of loss gradients as a function of distance in weight-space. We further investigate the importance of the true loss surface in neural networks and find, in contrast to previous work, that the exponential hardness of locating the global minimum has practical consequences for achieving state of the art performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑