本文构造了一个具有 Lipschitz 连续梯度的简单一维函数,证明了在无梯度噪声下,从原点出发的固定步长 ADAM 算法在最小化该函数时必然发散,且与参数选择无关。
摘要:
构造了一个具有 Lipschitz 连续梯度的极简单一维函数,使得在无梯度噪声下,从原点出发、采用固定步长的 ADAM 算法在最小化该函数时发生发散。无论方法参数如何选择,发散均会发生。
英文摘要:
A very simple unidimensional function with Lipschitz continuous gradient is constructed such that the ADAM algorithm with constant stepsize, started from the origin, diverges when applied to minimize this function in the absence of noise on the gradient. Divergence occurs irrespective of the choice of the method parameters.
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)(人工智能基础研究实验室(BRAIn实验室));Moscow Independent Research Institute of Artificial Intelligence(莫斯科人工智能独立研究院);Yandex Research(Yandex研究院);Institute of Science and Technology Austria (ISTA)(奥地利科学技术研究所(ISTA));Innopolis University(Innopolis大学)
Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
Peking University(北京大学);Beijing International Center for Mathematical Research, Peking University(北京大学北京国际数学研究中心);Center for Data Science, Peking University(北京大学数据科学中心);Beijing Institute of Big Data Research(北京大数据研究院)
一阶优化算法已被证明在深度学习中非常突出。特别是RMSProp和Adam等算法非常受欢迎。然而,最近的工作指出,缺乏“长期记忆“的亚当类算法,这可能会妨碍他们的性能,并导致分歧。在我们的研究中,我们观察到,在设计自适应学习率时,加权更多的过去梯度是有好处的。因此,我们提出了一种算法,称为怀旧亚当(NosAdam),理论上保证在最佳已知的收敛速度收敛。NosAdam可以被认为是对Adam的非收敛问题的修复,以替代[Reddi et al.,2018年]。我们的初步数值实验表明,NosAdam是一个有前途的替代算法亚当。证明,代码和其他补充材料可以在匿名共享链接中找到。
Analysis of Adam Algorithms for Stochastic Dynamic Systems
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(数学科学国家重点实验室,数学与系统科学研究院,中国科学院);School of Mathematical Sciences, University of Chinese Academy of Sciences(中国科学院大学数学科学学院);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院)
A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
Department of Mathematics, Stanford University(斯坦福大学数学系);Program in Applied and Computational Mathematics, Princeton University(普林斯顿大学应用与计算数学系);Department of Mathematics, Princeton University(普林斯顿大学数学系)
ZJU-UIUC Institute, Zhejiang University(浙江大学-伊利诺伊大学厄巴纳-香槟分校学院);Singapore University of Technology and Design(新加坡科技设计大学);Linköping University(林雪平大学);Princeton University(普林斯顿大学)
Department of Computer Science, University of Exeter(埃克塞特大学计算机科学系);Department of Mathematics and Computer Science, Eindhoven University of Technology(埃因霍温理工大学数学与计算机科学系);School of the Gifted Young, University of Science and Technology of China(中国科学技术大学天才青年学院);Department of Electrical and Computer Engineering, University of Texas at Austin(德克萨斯大学奥斯汀分校电气与计算机工程系);Department of Computer Science, University of Reading(阅读大学计算机科学系);School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络科学与技术学院);Department of Computer Science, The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校计算机科学系);ELLIS Institute Tubingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center, Tübingen, Germany(图宾根人工智能中心,德国图宾根);College of Computer Science, Chongqing University(重庆大学计算机学院)