Adam and RMSProp are two of the most influential adaptive stochastic algorithms for training deep neural networks, which have been pointed out to be divergent even in the convex setting via a few simple counterexamples. Many attempts, such as decreasing an adaptive learning rate, adopting a big batch size, incorporating a temporal decorrelation technique, seeking an analogous surrogate, etc., have been tried to promote Adam/RMSProp-type algorithms to converge. In contrast with existing approaches, we introduce an alternative easy-to-check sufficient condition, which merely depends on the parameters of the base learning rate and combinations of historical second-order moments, to guarantee the global convergence of generic Adam/RMSProp for solving large-scale non-convex stochastic optimization. Moreover, we show that the convergences of several variants of Adam, such as AdamNC, AdaEMA, etc., can be directly implied via the proposed sufficient condition in the non-convex setting. In addition, we illustrate that Adam is essentially a specifically weighted AdaGrad with exponential moving average momentum, which provides a novel perspective for understanding Adam and RMSProp. This observation coupled with this sufficient condition gives much deeper interpretations on their divergences. At last, we validate the sufficient condition by applying Adam and RMSProp to tackle a certain counterexample and train deep neural networks. Numerical results are exactly in accord with our theoretical analysis.
Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
Peking University(北京大学);Beijing International Center for Mathematical Research, Peking University(北京大学北京国际数学研究中心);Center for Data Science, Peking University(北京大学数据科学中心);Beijing Institute of Big Data Research(北京大数据研究院)
一阶优化算法已被证明在深度学习中非常突出。特别是RMSProp和Adam等算法非常受欢迎。然而,最近的工作指出,缺乏“长期记忆“的亚当类算法,这可能会妨碍他们的性能,并导致分歧。在我们的研究中,我们观察到,在设计自适应学习率时,加权更多的过去梯度是有好处的。因此,我们提出了一种算法,称为怀旧亚当(NosAdam),理论上保证在最佳已知的收敛速度收敛。NosAdam可以被认为是对Adam的非收敛问题的修复,以替代[Reddi et al.,2018年]。我们的初步数值实验表明,NosAdam是一个有前途的替代算法亚当。证明,代码和其他补充材料可以在匿名共享链接中找到。
Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
University of Basel(巴塞尔大学);University of Oslo(奥斯陆大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);ELLIS Institute Tübingen(埃利斯研究所蒂宾根分所);Tübingen AI Center(蒂宾根人工智能中心)
A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
Department of Mathematics, Stanford University(斯坦福大学数学系);Program in Applied and Computational Mathematics, Princeton University(普林斯顿大学应用与计算数学系);Department of Mathematics, Princeton University(普林斯顿大学数学系)
Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)(香港中文大学(深圳));Vietnam Academy of Science and Technology(越南科学技术研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院)
Adam revisited: a weighted past gradients perspective
University of Science and Technology of China(中国科学技术大学);Zhejiang Cainiao Supply Chain Management Co. Ltd(浙江菜鸟供应链管理有限公司);Advanced Digital Sciences Center(先进数字科学中心)
Moscow Institute of Physics and Technology(莫斯科物理技术学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Artificial Intelligence Research Institute(人工智能研究院);HSE University(高等经济大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学)
Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness
Department of Electrical and Computer Engineering, Queen’s University(皇后大学电气与计算机工程系);Department of Computer Science and Technology, University of Cambridge(剑桥大学计算机科学与技术系)
A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings
Robert Bosch Centre for Cyber-Physical Systems(罗伯特·博世网络物理系统中心);Indian Institute of Science(印度科学学院);Netaji Subhas University of Technology(纳塔吉·苏巴斯·技术大学);TCS Research(塔塔咨询服务公司研究中心)
本文研究非凸优化中 Adam 的固定步长收敛问题,推导并实现“精确”恒定步长,给出确定性与随机版本的收敛保证及运行时间界,并通过分类实验验证其降低梯度范数的效果。
Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Vietnam Academy of Science and Technology(越南科学技术研究院);Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)深圳大数据研究院)
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany(应用数学:分析与数值研究所,穆尔斯特大学,德国);School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China(数据科学学院和人工智能学院,香港中文大学(深圳)(CUHK-Shenzhen),中国)
Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks
Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates
University of Münster(明斯特大学);The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院)
Adaptive Gradient Methods with Dynamic Bound of Learning Rate
DiDi AI Labs(滴滴人工智能实验室);Peking University(北京大学);Zhejiang University(浙江大学);University of Southern California(南加州大学);Beijing Institute of Big Data Research(北京大数据研究院)
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
University of Minnesota - Twin Cities(明尼苏达大学双城分校);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research(IBM研究院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry
School of Computation, Information and Technology, Technical University of Munich(慕尼黑工业大学计算、信息与技术学院);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)