深度学习中阻尼的随机矩阵理论方法
A Random Matrix Theory Approach to Damping in Deep Learning
摘要:
我们推测,深度学习中自适应和非自适应梯度方法之间的内在差异源于真实损失表面最平坦方向上的估计噪声增加。我们证明了用于自适应方法(具有低数值稳定性或阻尼常数)的典型时间表有助于相对于尖锐方向向平坦方向偏移相对运动,有效地放大了信噪比并损害了泛化。我们进一步证明,在这些方法中使用的数值阻尼常数可以分解成一个学习率降低和线性收缩的估计曲率矩阵。然后,我们通过增加收缩系数来证明显着的泛化改进,在逻辑斯谛回归和几个深度神经网络实验中完全关闭泛化差距。进一步扩展这条线,我们开发了一种新的随机矩阵理论为基础的阻尼学习器的二阶优化的启发线性收缩估计。我们的实验表明,我们的学习器是非常不敏感的初始值,并允许与持续稳定的训练和竞争力的概括结合非常快的收敛。
英文摘要:
We conjecture that the inherent difference in generalisation between adaptive and non-adaptive gradient methods in deep learning stems from the increased estimation noise in the flattest directions of the true loss surface. We demonstrate that typical schedules used for adaptive methods (with low numerical stability or damping constants) serve to bias relative movement towards flat directions relative to sharp directions, effectively amplifying the noise-to-signal ratio and harming generalisation. We further demonstrate that the numerical damping constant used in these methods can be decomposed into a learning rate reduction and linear shrinkage of the estimated curvature matrix. We then demonstrate significant generalisation improvements by increasing the shrinkage coefficient, closing the generalisation gap entirely in both logistic regression and several deep neural network experiments. Extending this line further, we develop a novel random matrix theory based damping learner for second order optimiser inspired by linear shrinkage estimation. We experimentally demonstrate our learner to be very insensitive to the initialised value and to allow for extremely fast convergence in conjunction with continued stable training and competitive generalisation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。