Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $β_2$ near 1 for convergence, this work investigates the ``natural'' auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.
1-bit gradient compression and local steps are two representative techniques that enable drastic communication reduction in distributed SGD. Their benefits, however, remain an open question on Adam-based large model pre-training (e.g. BERT and GPT). In this paper, we demonstrate the non-linearity in Adam causes slow convergence even when 1-bit compression or local steps are individually applied. To alleviate this limitation, we propose 0/1 Adam that linearizes each Adam step via approximating its optimizer states using their stale estimates and linear correlation. 0/1 Adam performs an Adam-lik…
Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院，香港中文大学（深圳））；School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院和人工智能学院，香港中文大学（深圳））
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）；Peking University（北京大学）；Chinese Academy of Science（中国科学院）；CUHK (Shenzhen)（香港中文大学（深圳））
Department of Computer Science, The University of Iowa（爱荷华大学计算机科学系）；Machine Intelligence Technology, Alibaba Group（阿里巴巴集团机器智能技术）
Since its invention in 2014, the Adam optimizer has received tremendous attention. On one hand, it has been widely used in deep learning and many variants have been proposed, while on the other hand their theoretical convergence property remains to be a mystery. It is far from satisfactory in the sense that some studies require strong assumptions about the updates, which are not necessarily applicable in practice, while other studies still follow the original problematic convergence analysis of Adam, which was shown to be not sufficient to ensure convergence. Although rigorous convergence anal…
Microsoft（微软）；University of Rochester（罗切斯特大学）；ETH Zurich（苏黎世联邦理工学院）
Scalable training of large models (like BERT and GPT-3) requires careful optimization rooted in model design, architecture, and system capabilities. From a system standpoint, communication has become a major bottleneck, especially on commodity systems with standard TCP interconnects that offer limited network bandwidth. Communication compression is an important technique to reduce training time on such systems. One of the most effective methods is error-compensated compression, which offers robust convergence speed even under 1-bit compression. However, state-of-the-art error compensation tech…
We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with bounded gradients. We show that in expectation, the squared norm of the objective gradient averaged over the trajectory has an upper-bound which is explicit in the constants of the problem, parameters of the optimizer, the dimension $d$, and the total number of iterations $N$. This bound can be made arbitrarily small, and with the right hyper-parameters, Adam can be shown to converge with the same rate of convergence …
University of Amsterdam（阿姆斯特丹大学）；OpenAI；University of Toronto（多伦多大学）
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little…
A common way to train neural networks is the Backpropagation. This algorithm includes a gradient descent method, which needs an adaptive step size. In the area of neural networks, the ADAM-Optimizer is one of the most popular adaptive step size methods. It was invented in \cite{Kingma.2015} by Kingma and Ba. The $5865$ citations in only three years shows additionally the importance of the given paper. We discovered that the given convergence proof of the optimizer contains some mistakes, so that the proof will be wrong. In this paper we give an improvement to the convergence proof of the ADAM-…
Boston University（波士顿大学）；IBM T. J. Watson Research Center（IBM T. J. 沃森研究中心）；University of Iowa（爱荷华大学）
Adam is a widely used stochastic optimization method for deep learning applications. While practitioners prefer Adam because it requires less parameter tuning, its use is problematic from a theoretical point of view since it may not converge. Variants of Adam have been proposed with provable convergence guarantee, but they tend not be competitive with Adam on the practical performance. In this paper, we propose a new method named Adam$^+$ (pronounced as Adam-plus). Adam$^+$ retains some of the key components of Adam but it also has several noticeable differences: (i) it does not maintain the m…
Provable Adaptivity of Adam under Non-uniform Smoothness
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）；Chinese Academy of Mathematics and Systems Science（中国科学院数学与系统科学研究院）；Microsoft（微软）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit exa…
University of Minnesota - Twin Cities（明尼苏达大学双城分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as the Adam, AMSGrad and AdaGrad. Despite their popularity in training deep neural networks, the convergence of these algorithms for solving nonconvex problems remains an open question. This paper provides a set of mild sufficient conditions that guarantee the convergence for the Adam-type methods. We prove that under our derived conditions, these…
LTCI, Télécom Paris, Institut polytechnique de Paris（巴黎理工学院巴黎电信学院LTCI实验室）
Adam is a popular variant of stochastic gradient descent for finding a local minimizer of a function. In the constant stepsize regime, assuming that the objective function is differentiable and non-convex, we establish the convergence in the long run of the iterates to a stationary point under a stability condition. The key ingredient is the introduction of a continuous-time version of Adam, under the form of a non-autonomous ordinary differential equation. This continuous-time system is a relevant approximation of the Adam iterates, in the sense that the interpolated Adam process converges we…
Peking University（北京大学）；Beijing International Center for Mathematical Research, Peking University（北京大学北京国际数学研究中心）；Center for Data Science, Peking University（北京大学数据科学中心）；Beijing Institute of Big Data Research（北京大数据研究院）
First-order optimization algorithms have been proven prominent in deep learning. In particular, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of ``long-term memory" in Adam-like algorithms, which could hamper their performance and lead to divergence. In our study, we observe that there are benefits of weighting more of the past gradients when designing the adaptive learning rate. We therefore propose an algorithm called the Nostalgic Adam (NosAdam) with theoretically guaranteed convergence at the best known convergence rate. NosAdam …
Adaptive gradient methods (AGMs) have become popular in optimizing the nonconvex problems in deep learning area. We revisit AGMs and identify that the adaptive learning rate (A-LR) used by AGMs varies significantly across the dimensions of the problem over epochs (i.e., anisotropic scale), which may lead to issues in convergence and generalization. All existing modified AGMs actually represent efforts in revising the A-LR. Theoretically, we provide a new way to analyze the convergence of AGMs and prove that the convergence rate of \textsc{Adam} also depends on its hyper-parameter $ε$, which ha…
On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions
Center for Data Science and School of Mathematical Sciences, Zhejiang University（浙江大学数据科学中心与数学科学学院）；Center for Data Science, Zhejiang University（浙江大学数据科学中心）
Purdue University（普渡大学）；Sensetime Research（商汤科技研究院）；Chinese University of Hong Kong（香港中文大学）；University of Hong Kong（香港大学）
Although adaptive optimization algorithms such as Adam show fast convergence in many machine learning tasks, this paper identifies a problem of Adam by analyzing its performance in a simple non-convex synthetic problem, showing that Adam's fast convergence would possibly lead the algorithm to local minimums. To address this problem, we improve Adam by proposing a novel adaptive gradient descent algorithm named AdaX. Unlike Adam that ignores the past gradients, AdaX exponentially accumulates the long-term gradient information in the past during training, to adaptively tune the learning rate. We…
One of the most popular training algorithms for deep neural networks is the Adaptive Moment Estimation (Adam) introduced by Kingma and Ba. Despite its success in many applications there is no satisfactory convergence analysis: only local convergence can be shown for batch mode under some restrictions on the hyperparameters, counterexamples exist for incremental mode. Recent results show that for simple quadratic objective functions limit cycles of period 2 exist in batch mode, but only for atypical hyperparameters, and only for the algorithm without bias correction. %More general there are sev…
University of Maryland（马里兰大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the r…
Ton Duc Thang University（东德大学）；Meiji University（明治大学）；National Institute of Information and Communications Technology (NICT)（国立信息与通信技术研究所（NICT））
The adaptive moment estimation algorithm Adam (Kingma and Ba) is a popular optimizer in the training of deep neural networks. However, Reddi et al. have recently shown that the convergence proof of Adam is problematic and proposed a variant of Adam called AMSGrad as a fix. In this paper, we show that the convergence proof of AMSGrad is also problematic. Concretely, the problem in the convergence proof of AMSGrad is in handling the hyper-parameters, treating them as equal while they are not. This is also the neglected issue in the convergence proof of Adam. We provide an explicit counter-exampl…
This paper proposes a conjugate-gradient-based Adam algorithm blending Adam with nonlinear conjugate gradient methods and shows its convergence analysis. Numerical experiments on text classification and image classification show that the proposed algorithm can train deep neural network models in fewer epochs than the existing adaptive stochastic optimization algorithms can.
Changchun University of Science and Technology（长春理工大学）；Northeast Normal University（东北师范大学）
Adaptive gradient algorithm (AdaGrad) and its variants, such as RMSProp, Adam, AMSGrad, etc, have been widely used in deep learning. Although these algorithms are faster in the early phase of training, their generalization performance is often not as good as stochastic gradient descent (SGD). Hence, a trade-off method of transforming Adam to SGD after a certain iteration to gain the merits of both algorithms is theoretically and practically significant. To that end, we propose a decreasing scaling transition scheme to achieve a smooth and stable transition from Adam to SGD, which is called DST…
Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails
Department of Electrical and Computer Engineering, The Ohio State University（俄亥俄州立大学电气与计算机工程系）；School of Electrical, Computer and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
Moscow Institute of Physics and Technology, Russia（莫斯科物理技术学院）；Ivannikov Institute for System Programming RAS, Russia（伊万诺夫系统编程研究所）；Sber AI Lab, Russia（俄罗斯Sber人工智能实验室）；Innopolis University, Russia（因诺波利斯大学）；The Russian Presidential Academy of National Economy and Public Administration, Russia（俄罗斯总统经济与公共行政学院）；Skolkovo Institute of Science and Technology, Russia（斯克沃尔佐夫科学与技术研究所）；Mohamed bin Zayed University of Artificial Intelligence, UAE（穆罕默德·本·扎耶德人工智能大学）；Machine Learning and Optimization Laboratory (MLO), EPFL, Lausanne, Switzerland（机器学习与优化实验室（MLO），EPFL，拉瓦尔普勒斯，瑞士）
Convergence and convergence rate analyses of adaptive methods, such as Adaptive Moment Estimation (Adam) and its variants, have been widely studied for nonconvex optimization. The analyses are based on assumptions that the expected or empirical average loss function is Lipschitz smooth (i.e., its gradient is Lipschitz continuous) and the learning rates depend on the Lipschitz constant of the Lipschitz continuous gradient. Meanwhile, numerical evaluations of Adam and its variants have clarified that using small constant learning rates without depending on the Lipschitz constant and hyperparamet…
Adaptive Moment Estimation (ADAM) is a very popular training algorithm for deep neural networks and belongs to the family of adaptive gradient descent optimizers. However to the best of the authors knowledge no complete convergence analysis exists for ADAM. The contribution of this paper is a method for the local convergence analysis in batch mode for a deterministic fixed training set, which gives necessary conditions for the hyperparameters of the ADAM algorithm. Due to the local nature of the arguments the objective function can be non-convex but must be at least twice continuously differen…
An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning
1 Department of Computer Science \& Engineering, Indian Institute of Technology Indore, India.；3 National Remote Sensing Centre, Indian Space Research Organisation, India.
Microsoft（微软）；University of Rochester（罗切斯特大学）；ETH Zurich（苏黎世联邦理工学院）
Adam is the important optimization algorithm to guarantee efficiency and accuracy for training many important tasks such as BERT and ImageNet. However, Adam is generally not compatible with information (gradient) compression technology. Therefore, the communication usually becomes the bottleneck for parallelizing Adam. In this paper, we propose a communication efficient {\bf A}DAM {\bf p}reconditioned {\bf M}omentum SGD algorithm-- named APMSqueeze-- through an error compensated method compressing gradients. The proposed algorithm achieves a similar convergence efficiency to Adam in term of ep…
University of Maryland（马里兰大学）；Microsoft（微软）；Tsinghua University（清华大学）
Adaptive gradient methods such as RMSProp and Adam use exponential moving estimate of the squared gradient to compute adaptive step sizes, achieving better convergence than SGD in face of noisy objectives. However, Adam can have undesirable convergence behaviors due to unstable or extreme adaptive learning rates. Methods such as AMSGrad and AdaBound have been proposed to stabilize the adaptive learning rates of Adam in the later stage of training, but they do not outperform Adam in some practical tasks such as training Transformers \cite{transformer}. In this paper, we propose an adaptive lear…
The success of deep learning can be attributed to various factors such as increase in computational power, large datasets, deep convolutional neural networks, optimizers etc. Particularly, the choice of optimizer affects the generalization, convergence rate, and training stability. Stochastic Gradient Descent (SGD) is a first order iterative optimizer that updates the gradient uniformly for all parameters. This uniform update may not be suitable across the entire training phase. A rudimentary solution for this is to employ a fine-tuned learning rate scheduler which decreases learning rate as a…
University of California San Francisco（加州大学旧金山分校）；James Madison University（詹姆斯·麦迪逊大学）
In this work, to efficiently help escape the stationary and saddle points, we propose, analyze, and generalize a stochastic strategy performed as an operator for a first-order gradient descent algorithm in order to increase the target accuracy and reduce time consumption. Unlike existing algorithms, the proposed stochastic the strategy does not require any batches and sampling techniques, enabling efficient implementation and maintaining the initial first-order optimizer's convergence rate, but provides an incomparable improvement of target accuracy when optimizing the target functions. In sho…
LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI，巴黎高等电信学院，巴黎理工学院）
Although ADAM is a very popular algorithm for optimizing the weights of neural networks, it has been recently shown that it can diverge even in simple convex optimization examples. Several variants of ADAM have been proposed to circumvent this convergence issue. In this work, we study the ADAM algorithm for smooth nonconvex optimization under a boundedness assumption on the adaptive learning rate. The bound on the adaptive step size depends on the Lipschitz constant of the gradient of the objective function and provides safe theoretical adaptive step sizes. Under this boundedness assumption, w…
Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；Vietnam Academy of Science and Technology（越南科学技术研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Department of Computer Science, University of California, Los Angeles（加州大学洛杉矶分校计算机科学系）；School of Computing and Data Science, the University of Hong Kong（香港大学计算科学与数据科学学院）；Bytedance Inc（字节跳动公司）
提出 AdaGrad++ 和 Adam++ 两种简单无参数自适应梯度方法，在无需预设学习率的情况下实现与 AdaGrad 和 Adam 相当的收敛保证。
On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, China（机器人与自动信息系统研究所，人工智能学院，南开大学，天津，中国）；National Key Lab of General AI, School of Intelligence Science and Technology, Peking University, Beijing, China（通用人工智能国家重点实验室，智能科学与技术学院，北京大学，北京，中国）
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany（应用数学：分析与数值研究所，穆尔斯特大学，德国）；School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China（数据科学学院和人工智能学院，香港中文大学（深圳）（CUHK-Shenzhen），中国）
Center for Data Science（数据科学中心）；Peking University（北京大学）；Key Lab. of Machine Perception (MOE)（机器感知教育部重点实验室）；School of EECS, Peking University（北京大学电子工程与计算机科学学院）；Peng Cheng Laboratory（鹏城实验室）
Adaptive gradient methods, especially Adam-type methods (such as Adam, AMSGrad, and AdaBound), have been proposed to speed up the training process with an element-wise scaling term on learning rates. However, they often generalize poorly compared with stochastic gradient descent (SGD) and its accelerated schemes such as SGD with momentum (SGDM). In this paper, we propose a new adaptive method called DecGD, which simultaneously achieves good generalization like SGDM and obtain rapid convergence like Adam-type methods. In particular, DecGD decomposes the current gradient into the product of two …
University of Science and Technology of China（中国科学技术大学）；Zhejiang Cainiao Supply Chain Management Co. Ltd（浙江菜鸟供应链管理有限公司）；Advanced Digital Sciences Center（先进数字科学中心）
Adaptive learning rate methods have been successfully applied in many fields, especially in training deep neural networks. Recent results have shown that adaptive methods with exponential increasing weights on squared past gradients (i.e., ADAM, RMSPROP) may fail to converge to the optimal solution. Though many algorithms, such as AMSGRAD and ADAMNC, have been proposed to fix the non-convergence issues, achieving a data-dependent regret bound similar to or better than ADAGRAD is still a challenge to these methods. In this paper, we propose a novel adaptive method weighted adaptive algorithm (W…
A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning
Division of Data and Decision Sciences（数据与决策科学部）；Tata Consultancy Services Research（塔塔咨询服务公司研究院）；University of Maryland（马里兰大学）；Department of Mechanical Engineering（机械工程系）
本文基于状态空间建模与控制理论传递函数提出自适应梯度优化通用框架及 Adam 新变体 AdamSSM，证明其收敛性，并在图像分类和语言建模任务上验证其泛化与收敛优势。
In the context of stochastic gradient descent(SGD) and adaptive moment estimation (Adam),researchers have recently proposed optimization techniques that transition from Adam to SGD with the goal of improving both convergence and generalization performance. However, precisely how each approach trades off early progress and generalization is not well understood; thus, it is unclear when or even if, one should transition from one approach to the other. In this work, by first studying the convex setting, we identify potential contributors to observed differences in performance between SGD and Adam…
