This paper develops strong Gaussian approximations (KMT type) for constant learning rate stochastic gradient descent (SGD). We work with heavy tailed noise and assume that the number of iterates grows and the learning rate shrinks simultaneously. Our work constructs simple Gaussian processes on the same space and bounds, in probability, the maximum difference between the SGD errors and their respective limits. While there exist central limit theorems for constant rate SGD centered at the optimum, these only provide individual iterate inference. By contrast, our quenched KMT approximations allow for finite sample, simultaneous, whole-trajectory-level inference and imply stochastic process convergence (Functional Central Limit Theorems (FCLTs)). For accumulated errors centered at the true optimum, we first obtain a KMT-type approximation bound when the learning rate scales as $γ_n=\frac{c}{\sqrt n}+o(n^{-1/2})$ for a constant $c \in \mathbb{R}^+$ and $n$ iterations. The result reveals a trade off between controlling the bias of SGD and the initialization effect, with no learning rate scaling generally making both contributions vanish in the accumulated error limit. To eliminate these contributions, we consider a Burn in scheme and Richardson-Romberg extrapolation and prove respective KMT and FCLT theorems. We also prove a finite-horizon KMT-type approximation and FCLT for the rescaled iterate trajectory to a stationary Ornstein--Uhlenbeck (OU) process. We consider applications in risk control and simultaneous inference and perform simulations to illustrate our results.
Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate
Johns Hopkins University(约翰霍普金斯大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
理解\emph{stochastic gradient descent}(SGD)的算法偏见是现代机器学习和深度学习理论的关键挑战之一。然而,大多数现有的工作集中在\emph{very small or even infinitesimal}学习率制度,并未能涵盖实际的情况下,学习率是\emph{moderate and annealing}。在本文中,我们作出了初步的尝试,以描述特定的正则化效果的SGD在中等学习率制度,通过研究其行为优化过参数化线性回归问题。在这种情况下,SGD和GD是已知的收敛到唯一的最小范数的解决方案,然而,与温和的和退火学习率,我们表明,他们表现出不同的\emph{directional bias}:SGD收敛沿大特征值方向的数据矩阵,而GD后的小特征值方向。此外,我们表明,这样的方向性偏差的问题时,采用早期停止,其中SGD输出接近最优,但GD输出是次优的。最后,我们的理论解释了几个民间艺术在实践中用于SGD超参数调整,如(1)线性缩放初始学习率与批量大小;(2)超越SGD高学习率,即使损失停止下降。
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);New York University(纽约大学);Facebook AI Research(Facebook人工智能研究院);Politecnico di Torino(都灵理工大学);Microsoft Research New England(微软研究院新英格兰分部)
SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network
Texas A&M University(德克萨斯农工大学);Princeton University(普林斯顿大学);Massachusetts Institute of Technology(麻省理工学院);Center for Brains, Mind, and Machines(脑、心智与机器中心)
Bias-Aware Minimisation: Understanding and Mitigating Estimator Bias in Private SGD
Institute for AI in Medicine, Technical University of Munich(慕尼黑工业大学医学人工智能研究所);Konrad Zuse School for Excelllence in Reliable AI(康拉德·楚泽卓越可靠人工智能学院);V7 Labs(V7实验室);Imperial College London(帝国理工学院)
随机梯度下降(SGD)中的噪声为训练过参数化模型提供了至关重要的隐式正则化效应。以前的理论工作主要集中在球形高斯噪声,而经验研究表明,参数相关的噪声-由小批量或标签扰动引起的-比高斯噪声更有效。本文从理论上描述了这种现象的二次参数化模型介绍Vaskevicius等人。和Woodworth et el.我们表明,在一个过参数化的设置,SGD标签噪声恢复稀疏地面真理与任意初始化,而SGD高斯噪声或梯度下降过拟合到稠密的解决方案与大的规范。我们的分析表明,参数相关的噪声引入了一个偏向局部极小值较小的噪声方差,而球面高斯噪声没有。我们项目的代码是公开的。
Increasing Missingness to Reduce Bias: Richardson-SGD with Missing Data
Sorbonne Université and Université Paris Cité, CNRS, Laboratoire de Probabilités, Statistique et Modélisation, LPSM(索邦大学和巴黎cité大学,CNRS,概率、统计与建模实验室,LPSM)
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
Jagiellonian University(雅盖隆大学);Mila / University of Montreal(Mila / 蒙特利尔大学);Facebook AI Research(Facebook人工智能研究院);Ruhr University Bochum(波鸿鲁尔大学);University of Edinburgh(爱丁堡大学)
Risk Bounds of Accelerated SGD for Overparameterized Linear Regression
University of California, Los Angeles(加州大学洛杉矶分校);Simons Institute, University of California, Berkeley(加州大学伯克利分校西蒙斯研究所);Indiana University Bloomington(印第安纳大学伯明顿分校)
Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
University College London(伦敦大学学院);University of Cambridge(剑桥大学);MILA - Quebec AI institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Samsung - SAIT AI Lab Montreal(三星SAIT人工智能实验室蒙特利尔分部);Gatsby Computational Neuroscience Unit(加特兹计算神经科学中心);SWC - University College London(伦敦大学学院威尔科姆神经科学中心)
Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning
School of Mathematics and Statistics(数学与统计学学院);Guangxi Normal University(广西师范大学);School of Mathematics(数学学院);Hohai University(河海大学);College of Marine Life Sciences(海洋生命科学学院);Ocean University of China(中国海洋大学);State Key Laboratory of Mathematical Sciences(数学科学国家重点实验室);Academy of Mathematics and Systems Science(数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学)
Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning
NTT DATA Mathematical Systems Inc.(NTT数据数学系统公司);Graduate School of Information Science and Technology, The University of Tokyo(东京大学信息科学与技术研究生院);Center for Advanced Intelligence Project, RIKEN(理化学研究所先进智能项目中心)
在最近的集中式非凸分布式学习和联邦学习中,局部方法是减少通信时间的有前途的方法之一。然而,现有的工作主要集中在研究一阶最优性保证。另一方面,二阶最优性保证算法,即,逃逸鞍点的算法在非分布优化文献中已经被广泛研究。本文研究了一种新的局部算法--偏差方差降低局部扰动SGD(Bias-Variance Reduced Local Perturbed SGD,BVR-L-PSGD),它将已有的偏差方差降低梯度估计与参数扰动相结合,用于求解集中式非凸分布优化问题的二阶最优点。BVR-L-PSGD具有二阶最优性,但其通信复杂性与BVR-L-SGD的一阶最优性几乎相同。特别是当局部数据集的异质性小于局部损失的平滑性时,通信复杂性优于非局部方法。在极端情况下,当局部数据集的异质性为零时,通信复杂性度接近$\widetilde Θ(1)$。数值结果验证了我们的理论研究结果。
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Alan Turing Institute(艾伦·图灵研究所);University of Oxford(牛津大学);Florida State University(佛罗里达州立大学);Rutgers Business School(罗格斯商学院);INRIA(法国国家信息与自动化研究所);École Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学)
Beyond Optimal Rates in Stochastic Optimization: Trajectory-Adaptive Stopping Rules
UC Berkeley(加州大学伯克利分校);Inria(法国国家信息与自动化研究所);École Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学);Department of Statistics, UC Berkeley(加州大学伯克利分校统计系)
College of Computer and Control Engineering, Nankai University(南开大学计算机与控制工程学院);MILA(米拉人工智能研究所);Université de Montréal(蒙特利尔大学);Aristotle University of Thessaloniki(塞萨洛尼基亚里士多德大学)
Reducing Bias in Deep Learning Optimization: The RSGDM Approach
Stevens Institute of Technology(史蒂文斯理工学院);University of California, Berkeley(加州大学伯克利分校);Chinese University of Hong Kong(香港中文大学);University of California, Irvine(加州大学欧文分校);Illinois Institute of Technology(伊利诺伊理工学院);Southern Methodist University(南卫理公会大学)
Quebec Artificial Intelligence Institute (Mila)(魁北克人工智能研究所(米拉));Université de Montréal(蒙特利尔大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Athens(雅典大学)