arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1811.09358cs.LGcs.CVcs.NAmath.NAmath.OCstat.ML
收藏

Adam和RMSProp收敛性的充分条件

A Sufficient Condition for Convergences of Adam and RMSProp

Tencent AI Lab(腾讯AI实验室);Stony Brook University(石英布鲁克大学)

Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, Wei Liu

AI总结:

本文提出了一种易于检查的充分条件,该条件仅依赖于基础学习率参数和历史二阶矩量的组合,以保证通用的Adam/RMSProp算法在大规模非凸随机优化中的全局收敛性,并展示了几种Adam变体在非凸设置下的收敛性可由此条件直接推导。

摘要:

Adam和RMSProp是训练深度神经网络中最具影响力的自适应随机算法,尽管在凸设置中通过几个简单的反例已被指出存在发散现象。许多尝试,如降低自适应学习率、采用大批次大小、引入时间去相关技术、寻找类比的替代方案等,已被尝试以促进Adam/RMSProp型算法收敛。与现有方法不同,我们引入了一种替代的易于检查的充分条件,该条件仅依赖于基础学习率参数和历史二阶矩量的组合,以保证通用的Adam/RMSProp算法在大规模非凸随机优化中的全局收敛性。此外,我们展示了几种Adam变体,如AdamNC、AdaEMA等,在非凸设置下的收敛性可通过所提出的充分条件直接推导。此外,我们表明Adam本质上是一种具有指数移动平均动量的特定加权AdaGrad,这为理解Adam和RMSProp提供了新的视角。这一观察结合该充分条件,为它们的发散性提供了更深入的解释。最后,我们通过将Adam和RMSProp应用于特定反例和训练深度神经网络来验证该充分条件。数值结果与我们的理论分析一致。

英文摘要:

Adam and RMSProp are two of the most influential adaptive stochastic algorithms for training deep neural networks, which have been pointed out to be divergent even in the convex setting via a few simple counterexamples. Many attempts, such as decreasing an adaptive learning rate, adopting a big batch size, incorporating a temporal decorrelation technique, seeking an analogous surrogate, etc., have been tried to promote Adam/RMSProp-type algorithms to converge. In contrast with existing approaches, we introduce an alternative easy-to-check sufficient condition, which merely depends on the parameters of the base learning rate and combinations of historical second-order moments, to guarantee the global convergence of generic Adam/RMSProp for solving large-scale non-convex stochastic optimization. Moreover, we show that the convergences of several variants of Adam, such as AdamNC, AdaEMA, etc., can be directly implied via the proposed sufficient condition in the non-convex setting. In addition, we illustrate that Adam is essentially a specifically weighted AdaGrad with exponential moving average momentum, which provides a novel perspective for understanding Adam and RMSProp. This observation coupled with this sufficient condition gives much deeper interpretations on their divergences. At last, we validate the sufficient condition by applying Adam and RMSProp to tackle a certain counterexample and train deep neural networks. Numerical results are exactly in accord with our theoretical analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑