This paper proposes a conjugate-gradient-based Adam algorithm blending Adam with nonlinear conjugate gradient methods and shows its convergence analysis. Numerical experiments on text classification and image classification show that the proposed algorithm can train deep neural network models in fewer epochs than the existing adaptive stochastic optimization algorithms can.
University of Science and Technology of China（中国科学技术大学）；Zhejiang Cainiao Supply Chain Management Co. Ltd（浙江菜鸟供应链管理有限公司）；Advanced Digital Sciences Center（先进数字科学中心）
Adaptive learning rate methods have been successfully applied in many fields, especially in training deep neural networks. Recent results have shown that adaptive methods with exponential increasing weights on squared past gradients (i.e., ADAM, RMSPROP) may fail to converge to the optimal solution. Though many algorithms, such as AMSGRAD and ADAMNC, have been proposed to fix the non-convergence issues, achieving a data-dependent regret bound similar to or better than ADAGRAD is still a challenge to these methods. In this paper, we propose a novel adaptive method weighted adaptive algorithm (W…
Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院，香港中文大学（深圳））；Department of Probability and Statistic, Institute of Mathematics, Vietnam Academy of Science and Technology（概率与统计系，数学研究所，越南科学与技术 academy）；School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院和人工智能学院，香港中文大学（深圳））
Center for Data Science（数据科学中心）；Peking University（北京大学）；Key Lab. of Machine Perception (MOE)（机器感知教育部重点实验室）；School of EECS, Peking University（北京大学电子工程与计算机科学学院）；Peng Cheng Laboratory（鹏城实验室）
Adaptive gradient methods, especially Adam-type methods (such as Adam, AMSGrad, and AdaBound), have been proposed to speed up the training process with an element-wise scaling term on learning rates. However, they often generalize poorly compared with stochastic gradient descent (SGD) and its accelerated schemes such as SGD with momentum (SGDM). In this paper, we propose a new adaptive method called DecGD, which simultaneously achieves good generalization like SGDM and obtain rapid convergence like Adam-type methods. In particular, DecGD decomposes the current gradient into the product of two …
Rensselaer Polytechnic Institute（伦斯勒理工学院）；University of California, Los Angeles（加州大学洛杉矶分校）
Stochastic gradient descent (SGD) has taken the stage as the primary workhorse for large-scale machine learning. It is often used with its adaptive variants such as AdaGrad, Adam, and AMSGrad. This paper proposes an adaptive stochastic gradient descent method for distributed machine learning, which can be viewed as the communication-adaptive counterpart of the celebrated Adam method - justifying its name CADA. The key components of CADA are a set of new rules tailored for adaptive stochastic gradients that can be implemented to save communication upload. The new algorithms adaptively reuse the…
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）；Peking University（北京大学）；Chinese Academy of Science（中国科学院）；CUHK (Shenzhen)（香港中文大学（深圳））
Adaptive gradient methods have shown excellent performances for solving many machine learning problems. Although multiple adaptive gradient methods were recently studied, they mainly focus on either empirical or theoretical aspects and also only work for specific problems by using some specific adaptive learning rates. Thus, it is desired to design a universal framework for practical algorithms of adaptive gradients with theoretical guarantee to solve general problems. To fill this gap, we propose a faster and universal framework of adaptive gradients (i.e., SUPER-ADAM) by introducing a univer…
Salesforce Research（Salesforce研究院）；National University of Singapore（新加坡国立大学）；Princeton University（普林斯顿大学）
It is not clear yet why ADAM-alike adaptive gradient algorithms suffer from worse generalization performance than SGD despite their faster training speed. This work aims to provide understandings on this generalization gap by analyzing their local convergence behaviors. Specifically, we observe the heavy tails of gradient noise in these algorithms. This motivates us to analyze these algorithms through their Levy-driven stochastic differential equations (SDEs) because of the similar convergence behaviors of an algorithm and its SDE. Then we establish the escaping time of these SDEs from a local…
Convergence and convergence rate analyses of adaptive methods, such as Adaptive Moment Estimation (Adam) and its variants, have been widely studied for nonconvex optimization. The analyses are based on assumptions that the expected or empirical average loss function is Lipschitz smooth (i.e., its gradient is Lipschitz continuous) and the learning rates depend on the Lipschitz constant of the Lipschitz continuous gradient. Meanwhile, numerical evaluations of Adam and its variants have clarified that using small constant learning rates without depending on the Lipschitz constant and hyperparamet…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an estimate of their relative variance. We disentangle these two aspects and analyze them in isolation, gaining insight into the mechanisms underlying ADAM. This analysis also extends recent results on adverse effects of ADAM on generalization, isolating the sign aspect as the problemat…
Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院，香港中文大学（深圳））；School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院和人工智能学院，香港中文大学（深圳））
University of Minnesota（明尼苏达大学）；MIT-IBM Watson AI Lab, IBM Research（麻省理工学院-IBM沃森人工智能实验室，IBM研究院）；Northeastern University（东北大学）；Princeton University（普林斯顿大学）
The adaptive momentum method (AdaMM), which uses past gradients to update descent directions and learning rates simultaneously, has become one of the most popular first-order optimization methods for solving machine learning problems. However, AdaMM is not suited for solving black-box optimization problems, where explicit gradient forms are difficult or infeasible to obtain. In this paper, we propose a zeroth-order AdaMM (ZO-AdaMM) algorithm, that generalizes AdaMM to the gradient-free regime. We show that the convergence rate of ZO-AdaMM for both convex and nonconvex optimization is roughly a…
Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well in the initial portion of training but are outperformed by SGD at later stages of training. We investigate a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate. Concretely, we propose SWATS, a simple strategy which switches from Adam to SGD when a triggering condition is satisfied. The condition we propose relates to the projection o…
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit exa…
A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning
Division of Data and Decision Sciences（数据与决策科学部）；Tata Consultancy Services Research（塔塔咨询服务公司研究院）；University of Maryland（马里兰大学）；Department of Mechanical Engineering（机械工程系）
本文基于状态空间建模与控制理论传递函数提出自适应梯度优化通用框架及 Adam 新变体 AdamSSM，证明其收敛性，并在图像分类和语言建模任务上验证其泛化与收敛优势。
University of Minnesota - Twin Cities（明尼苏达大学双城分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as the Adam, AMSGrad and AdaGrad. Despite their popularity in training deep neural networks, the convergence of these algorithms for solving nonconvex problems remains an open question. This paper provides a set of mild sufficient conditions that guarantee the convergence for the Adam-type methods. We prove that under our derived conditions, these…
Purdue University（普渡大学）；Sensetime Research（商汤科技研究院）；Chinese University of Hong Kong（香港中文大学）；University of Hong Kong（香港大学）
Although adaptive optimization algorithms such as Adam show fast convergence in many machine learning tasks, this paper identifies a problem of Adam by analyzing its performance in a simple non-convex synthetic problem, showing that Adam's fast convergence would possibly lead the algorithm to local minimums. To address this problem, we improve Adam by proposing a novel adaptive gradient descent algorithm named AdaX. Unlike Adam that ignores the past gradients, AdaX exponentially accumulates the long-term gradient information in the past during training, to adaptively tune the learning rate. We…
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
Moscow Institute of Physics and Technology, Russia（莫斯科物理技术学院）；Ivannikov Institute for System Programming RAS, Russia（伊万诺夫系统编程研究所）；Sber AI Lab, Russia（俄罗斯Sber人工智能实验室）；Innopolis University, Russia（因诺波利斯大学）；The Russian Presidential Academy of National Economy and Public Administration, Russia（俄罗斯总统经济与公共行政学院）；Skolkovo Institute of Science and Technology, Russia（斯克沃尔佐夫科学与技术研究所）；Mohamed bin Zayed University of Artificial Intelligence, UAE（穆罕默德·本·扎耶德人工智能大学）；Machine Learning and Optimization Laboratory (MLO), EPFL, Lausanne, Switzerland（机器学习与优化实验室（MLO），EPFL，拉瓦尔普勒斯，瑞士）
University of California, Los Angeles（加州大学洛杉矶分校）；The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）
Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient descent, Adam can converge to a different solution with a significantly worse test error in many deep learning applications such as image classification, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for …
Peking University（北京大学）；Beijing International Center for Mathematical Research, Peking University（北京大学北京国际数学研究中心）；Center for Data Science, Peking University（北京大学数据科学中心）；Beijing Institute of Big Data Research（北京大数据研究院）
First-order optimization algorithms have been proven prominent in deep learning. In particular, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of ``long-term memory" in Adam-like algorithms, which could hamper their performance and lead to divergence. In our study, we observe that there are benefits of weighting more of the past gradients when designing the adaptive learning rate. We therefore propose an algorithm called the Nostalgic Adam (NosAdam) with theoretically guaranteed convergence at the best known convergence rate. NosAdam …
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany（应用数学：分析与数值研究所，穆尔斯特大学，德国）；School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China（数据科学学院和人工智能学院，香港中文大学（深圳）（CUHK-Shenzhen），中国）
Department of Computer Science, The University of Iowa（爱荷华大学计算机科学系）；Machine Intelligence Technology, Alibaba Group（阿里巴巴集团机器智能技术）
Since its invention in 2014, the Adam optimizer has received tremendous attention. On one hand, it has been widely used in deep learning and many variants have been proposed, while on the other hand their theoretical convergence property remains to be a mystery. It is far from satisfactory in the sense that some studies require strong assumptions about the updates, which are not necessarily applicable in practice, while other studies still follow the original problematic convergence analysis of Adam, which was shown to be not sufficient to ensure convergence. Although rigorous convergence anal…
University of Maryland（马里兰大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the r…
University of California San Francisco（加州大学旧金山分校）；James Madison University（詹姆斯·麦迪逊大学）
In this work, to efficiently help escape the stationary and saddle points, we propose, analyze, and generalize a stochastic strategy performed as an operator for a first-order gradient descent algorithm in order to increase the target accuracy and reduce time consumption. Unlike existing algorithms, the proposed stochastic the strategy does not require any batches and sampling techniques, enabling efficient implementation and maintaining the initial first-order optimizer's convergence rate, but provides an incomparable improvement of target accuracy when optimizing the target functions. In sho…
University of Amsterdam（阿姆斯特丹大学）；OpenAI；University of Toronto（多伦多大学）
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little…
Northeastern University（东北大学）；University of California, Davis（加利福尼亚大学戴维斯分校）
Adaptive gradient methods, e.g. \textsc{Adam}, have achieved tremendous success in machine learning. Scaling the learning rate element-wisely by a certain form of second moment estimate of gradients, such methods are able to attain rapid training of modern deep neural networks. Nevertheless, they are observed to suffer from compromised generalization ability compared with stochastic gradient descent (\textsc{SGD}) and tend to be trapped in local minima at an early stage during training. Intriguingly, we discover that substituting the gradient in the second raw moment estimate term with its mom…
How do simple rotations affect the implicit bias of Adam?
Committee on Computational and Applied Mathematics, University of Chicago（计算与应用数学委员会，芝加哥大学）；Department of Electrical & Computer Engineering, University of Washington（电气与计算机工程系，华盛顿大学）；NSF-Simons National Institute for Theory and Mathematics in Biology（NSF-西蒙斯国家理论与生物数学研究所）；Department of Statistics, Department of Computer Science, and Data Science Institute, University of Chicago（统计系、计算机科学系和数据科学研究所，芝加哥大学）
Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Department of Computer Science, University of California, Los Angeles（加州大学洛杉矶分校计算机科学系）；School of Computing and Data Science, the University of Hong Kong（香港大学计算科学与数据科学学院）；Bytedance Inc（字节跳动公司）
提出 AdaGrad++ 和 Adam++ 两种简单无参数自适应梯度方法，在无需预设学习率的情况下实现与 AdaGrad 和 Adam 相当的收敛保证。
University of Maryland（马里兰大学）；Microsoft（微软）；Tsinghua University（清华大学）
Adaptive gradient methods such as RMSProp and Adam use exponential moving estimate of the squared gradient to compute adaptive step sizes, achieving better convergence than SGD in face of noisy objectives. However, Adam can have undesirable convergence behaviors due to unstable or extreme adaptive learning rates. Methods such as AMSGrad and AdaBound have been proposed to stabilize the adaptive learning rates of Adam in the later stage of training, but they do not outperform Adam in some practical tasks such as training Transformers \cite{transformer}. In this paper, we propose an adaptive lear…
Changchun University of Science and Technology（长春理工大学）；Northeast Normal University（东北师范大学）
Adaptive gradient algorithm (AdaGrad) and its variants, such as RMSProp, Adam, AMSGrad, etc, have been widely used in deep learning. Although these algorithms are faster in the early phase of training, their generalization performance is often not as good as stochastic gradient descent (SGD). Hence, a trade-off method of transforming Adam to SGD after a certain iteration to gain the merits of both algorithms is theoretically and practically significant. To that end, we propose a decreasing scaling transition scheme to achieve a smooth and stable transition from Adam to SGD, which is called DST…
On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions
Center for Data Science and School of Mathematical Sciences, Zhejiang University（浙江大学数据科学中心与数学科学学院）；Center for Data Science, Zhejiang University（浙江大学数据科学中心）
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam or AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.
University of Padua（帕多瓦大学）；Università di Bologna（博洛尼亚大学）
Stochastic gradient descent (SGD) is the main approach for training deep networks: it moves towards the optimum of the cost function by iteratively updating the parameters of a model in the direction of the gradient of the loss evaluated on a minibatch. Several variants of SGD have been proposed to make adaptive step sizes for each parameter (adaptive gradient) and take into account the previous updates (momentum). Among several alternative of SGD the most popular are AdaGrad, AdaDelta, RMSProp and Adam which scale coordinates of the gradient by square roots of some form of averaging of the sq…
ELLIS Institute Tübingen（图宾根埃利斯研究所）；MPI-IS Tübingen（图宾根马克斯·普朗克研究所）；AI Center, Germany（德国人工智能中心）；CCM, Flatiron Institute, Simons Foundation（CCM，Flatiron研究所，Simons基金会）；New York, US（美国纽约）
