快速自适应矩估计
Rapidly Adapting Moment Estimation
摘要：
自适应梯度方法（如Adam）已被证明通过跟踪梯度的二阶矩来计算个体学习率，对训练深度神经网络（DNN）非常有效。与现有的方法不同，我们利用最近的梯度的一阶矩来计算每次迭代的个体学习率。其背后的动机是梯度的一阶矩的动态变化可以提供有用的信息来获得学习率。我们称之为快速自适应矩估计（RAME）的新方法。本文用类似于[1]中对Adam的分析方法研究了确定性RAME的理论收敛性。训练一些DNN的实验结果表明，RAME w.r.t.的收敛速度和推广性能相比，随机重球（SHB）的方法，亚当，和RMSprop。
英文摘要：
Adaptive gradient methods such as Adam have been shown to be very effective for training deep neural networks (DNNs) by tracking the second moment of gradients to compute the individual learning rates. Differently from existing methods, we make use of the most recent first moment of gradients to compute the individual learning rates per iteration. The motivation behind it is that the dynamic variation of the first moment of gradients may provide useful information to obtain the learning rates. We refer to the new method as the rapidly adapting moment estimation (RAME). The theoretical convergence of deterministic RAME is studied by using an analysis similar to the one used in [1] for Adam. Experimental results for training a number of DNNs show promising performance of RAME w.r.t. the convergence speed and generalization performance compared to the stochastic heavy-ball (SHB) method, Adam, and RMSprop.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
