Adam is the default algorithm for training neural networks, including large language models (LLMs). However, \citet{reddi2019convergence} provided an example that Adam diverges, raising concerns for its deployment in AI model training. We identify a key mismatch between the divergence example and practice: \citet{reddi2019convergence} pick the problem after picking the hyperparameters of Adam, i.e., $(β_1,β_2)$; while practical applications often fix the problem first and then tune $(β_1,β_2)$. In this work, we prove that Adam converges with proper problem-dependent hyperparameters. First, we prove that Adam converges when $β_2$ is large and $β_1 < \sqrt{β_2}$. Second, when $β_2$ is small, we point out a region of $(β_1,β_2)$ combinations where Adam can diverge to infinity. Our results indicate a phase transition for Adam from divergence to convergence when changing the $(β_1, β_2)$ combination. To our knowledge, this is the first phase transition in $(β_1,β_2)$ 2D-plane reported in the literature, providing rigorous theoretical guarantees for Adam optimizer. We further point out that the critical boundary $(β_1^*, β_2^*)$ is problem-dependent, and particularly, dependent on batch size. This provides suggestions on how to tune $β_1$ and $β_2$: when Adam does not work well, we suggest tuning up $β_2$ inversely with batch size to surpass the threshold $β_2^*$, and then trying $β_1< \sqrt{β_2}$. Our suggestions are supported by reports from several empirical studies, which observe improved LLM training performance when applying them.
Provable Adaptivity of Adam under Non-uniform Smoothness
University of Science and Technology of China(中国科学技术大学);Microsoft Research Asia(微软亚洲研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Peking University(北京大学);Chinese Academy of Mathematics and Systems Science(中国科学院数学与系统科学研究院);Microsoft(微软);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees
Institute of Operations Research and Analytics(运筹学与分析研究所);National University of Singapore(新加坡国立大学);School of Computer and Computing Science(计算机与计算科学学院);Hangzhou City University(杭州城市大学);State Key Laboratory of Scientific and Engineering Computing(科学与工程计算国家重点实验室);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Department of Mathematics(数学系)
Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM
University of Connecticut(康涅狄格大学)
自适应梯度方法(AGM)在深度学习领域的非凸优化问题中得到了广泛的应用。我们重新审视AGM,并确定AGM使用的自适应学习率(A-LR)在不同时期的问题维度上变化很大(即,各向异性尺度),这可能导致收敛和泛化方面的问题。所有现有的修改后年度股东大会实际上都代表了修改A-LR的努力。在理论上,我们提供了一种新的方法来分析AGM的收敛性,并证明了\textsc{Adam}的收敛速度也依赖于它的超参数$ε$,这是以前被忽视的。基于这两个事实,我们提出了一个新的AGM校准A-LR与激活({ \em softplus})功能,导致\textsc{Sadam}和\textsc{SAMSGrad}方法\footnote{Code is available at https://github.com/neilliang90/Sadam.git.}。我们进一步证明了这些算法在非凸、非强凸和Polyak-Projojasiewicz条件下比\textsc{Adam}具有更好的收敛速度。实证研究支持我们对各向异性A-LR的观察,并表明所提出的方法优于现有的AGM,并且在多个深度学习任务中比S-Momentum更好地推广。
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
Moscow Institute of Physics and Technology, Russia(莫斯科物理技术学院);Ivannikov Institute for System Programming RAS, Russia(伊万诺夫系统编程研究所);Sber AI Lab, Russia(俄罗斯Sber人工智能实验室);Innopolis University, Russia(因诺波利斯大学);The Russian Presidential Academy of National Economy and Public Administration, Russia(俄罗斯总统经济与公共行政学院);Skolkovo Institute of Science and Technology, Russia(斯克沃尔佐夫科学与技术研究所);Mohamed bin Zayed University of Artificial Intelligence, UAE(穆罕默德·本·扎耶德人工智能大学);Machine Learning and Optimization Laboratory (MLO), EPFL, Lausanne, Switzerland(机器学习与优化实验室(MLO),EPFL,拉瓦尔普勒斯,瑞士)
Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity
University of Science and Technology of China(中国科学技术大学);Microsoft Research(微软研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization
University of Minnesota(明尼苏达大学);MIT-IBM Watson AI Lab, IBM Research(麻省理工学院-IBM沃森人工智能实验室,IBM研究院);Northeastern University(东北大学);Princeton University(普林斯顿大学)
Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany(应用数学:分析与数值研究所,穆尔斯特大学,德国);School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China(数据科学学院和人工智能学院,香港中文大学(深圳)(CUHK-Shenzhen),中国)
Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
School of Mathematical Sciences, Peking University(北京大学数学科学学院);Department of Computer Science, University of California, Los Angeles(加州大学洛杉矶分校计算机科学系);School of Computing and Data Science, the University of Hong Kong(香港大学计算科学与数据科学学院);Bytedance Inc(字节跳动公司)
提出 AdaGrad++ 和 Adam++ 两种简单无参数自适应梯度方法,在无需预设学习率的情况下实现与 AdaGrad 和 Adam 相当的收敛保证。
Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
Peking University(北京大学);Beijing International Center for Mathematical Research, Peking University(北京大学北京国际数学研究中心);Center for Data Science, Peking University(北京大学数据科学中心);Beijing Institute of Big Data Research(北京大数据研究院)
一阶优化算法已被证明在深度学习中非常突出。特别是RMSProp和Adam等算法非常受欢迎。然而,最近的工作指出,缺乏“长期记忆“的亚当类算法,这可能会妨碍他们的性能,并导致分歧。在我们的研究中,我们观察到,在设计自适应学习率时,加权更多的过去梯度是有好处的。因此,我们提出了一种算法,称为怀旧亚当(NosAdam),理论上保证在最佳已知的收敛速度收敛。NosAdam可以被认为是对Adam的非收敛问题的修复,以替代[Reddi et al.,2018年]。我们的初步数值实验表明,NosAdam是一个有前途的替代算法亚当。证明,代码和其他补充材料可以在匿名共享链接中找到。
Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers
School of Information Science and Engineering, Lanzhou University(兰州大学信息科学与工程学院);Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology(新泽西理工学院电气与计算机工程系)
Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates
University of Münster(明斯特大学);The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院)
Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)(香港中文大学(深圳));Vietnam Academy of Science and Technology(越南科学技术研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院)
Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Vietnam Academy of Science and Technology(越南科学技术研究院);Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)深圳大数据研究院)
Conda: Column-Normalized Adam for Training Large Language Models Faster
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学院,北京大学);Singapore Management University(新加坡国立大学);Nankai University(南开大学);Beijing Normal University(北京师范大学);ByteDance Seed(字节跳动种子);Beijing University of Posts and Telecommunications(北京邮电大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学);Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China(黄埔实验室(广州),广东,中国)
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training
Wangxuan Institute of Computer Technology, Peking University(北京大学王宣计算机技术研究所);University of Science and Technology of China(中国科学技术大学);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室)
Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
School of Data Science and Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)(数据科学学院和深圳大数据研究院,香港中文大学(深圳))
A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
Department of Mathematics, Stanford University(斯坦福大学数学系);Program in Applied and Computational Mathematics, Princeton University(普林斯顿大学应用与计算数学系);Department of Mathematics, Princeton University(普林斯顿大学数学系)
Promoting Exploration in Memory-Augmented Adam using Critical Momenta
Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Polytechnique Montreal(蒙特利尔理工学院);Samsung SAIT AI Lab Montreal(三星SAIT人工智能实验室蒙特利尔分部);Université de Montréal(蒙特利尔大学);Google DeepMind(谷歌DeepMind)