We study the problem of solving strongly convex and smooth unconstrained optimization problems using stochastic first-order algorithms. We devise a novel algorithm, referred to as Recursive One-Over-T SGD (ROOT-SGD), based on an easily implementable, recursive averaging of past stochastic gradients. We prove that it simultaneously achieves state-of-the-art performance in both a finite-sample, nonasymptotic sense and an asymptotic sense. On the non-asymptotic side, we prove risk bounds on the last iterate of ROOT-SGD with leading-order terms that match the optimal statistical risk with a unity pre-factor, along with a higher-order term that scales at the sharp rate of $O(n^{-3/2})$ under the Lipschitz condition on the Hessian matrix. On the asymptotic side, we show that when a mild, one-point Hessian continuity condition is imposed, the rescaled last iterate of (multi-epoch) ROOT-SGD converges asymptotically to a Gaussian limit with the Cramér-Rao optimal asymptotic covariance, for a broad range of step-size choices.
Nonasymptotic Analysis of Stochastic Gradient Descent with the Richardson-Romberg Extrapolation
HSE University(俄罗斯高等经济大学);Duisburg-Essen University(杜伊斯堡-埃森大学);CMAP, UMR 7641, Ecole Polytechnique(巴黎高等理工学院CMAP研究所);Mohamed Bin Zayed University of AI(马尔代夫人工智能大学);Skolkovo Institute of Science and Technology(斯克尔科沃科技研究院);Steklov Mathematical Institute of Russian Academy of Sciences(俄罗斯科学院斯克洛夫数学研究所)
Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent
Inria Paris(巴黎国立信息与自动化研究所);CNRS(法国国家科学研究中心);Ecole Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学);CMAP(应用数学中心);Ecole Polytechnique(巴黎综合理工学院);Institut Polytechnique de Paris(巴黎理工学院)
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
Institut de Physique Théorique, CNRS, CEA, Université Paris-Saclay(巴黎萨克雷大学、法国国家科学研究中心、原子能和替代能源委员会理论物理研究所);Center for Brain Science, Harvard University(哈佛大学脑科学中心);Department of Experimental Psychology, University of Oxford(牛津大学实验心理学系)
Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms
Alfréd Rényi Institute of Mathematics(阿尔弗雷德·雷尼数学研究所);The University of Edinburgh(爱丁堡大学);The Alan Turing Institute(艾伦·图灵研究所);National Technical University of Athens(雅典国立技术大学)
HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization
College of Computer Science and Technology(计算机科学与技术学院);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);MIIT Key Laboratory of Pattern Analysis and Machine Intelligence(模式分析与机器智能重点实验室)
Mathematics and Statistics Department, McGill University(麦吉尔大学数学与统计学系);Mathematics and Statistics Department, Université de Montréal(蒙特利尔大学数学与统计学系);Google DeepMind(谷歌DeepMind)