High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise
Carnegie Mellon University(卡内基梅隆大学);Faculty of Technical Sciences, University of Novi Sad(诺维萨德大学技术科学学院);Faculty of Sciences, University of Novi Sad(诺维萨德大学理学院)
Aleksandar Armacki, Pranay Sharma, Gauri Joshi, Dragana Bajovic, Dusan Jakovetic, Soummya Kar
We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-ζ} \right)$, where $ζ\in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $ζ$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $η\in (1,2]$, and establish convergence rates whose exponents go to zero as $η\rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.
Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clipping
Inria, CNRS, Ecole Normale Supérieure PSL Research University(法国国家科学研究中心、巴黎高等师范学院PSL研究大学);Department of Management Science and Information Systems(管理科学与信息系统系);Faculty of Engineering and Natural Sciences(工程与自然科学学院);Department of Mathematics(数学系)
Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descent
Nordita(北欧理论物理研究所);KTH Royal Institute of Technology(瑞典皇家理工学院);Stockholm University(斯德哥尔摩大学);École Normale Supérieure, Université PSL(巴黎文理研究大学高等师范学院);CNRS(法国国家科学研究中心);INRIA(法国国家信息与自动化研究所)
Accelerated stochastic first-order method for convex optimization under heavy-tailed noise
Department of Mathematics, Linköping University, Sweden(林雪平大学数学系);Department of Industrial and Systems Engineering, University of Minnesota, USA(明尼苏达大学工业与系统工程系)
Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
KAUST Center of Excellence for Generative AI(知识增强人工智能卓越中心);ETH Zurich(苏黎世联邦理工学院);ETH AI Center(苏黎世联邦理工学院人工智能中心);Georgia Institute of Technology(佐治亚理工学院)
Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Inria(法国国家信息与自动化研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学);Boğaziçi University(博阿齐奇大学);Rutgers University(罗格斯大学);Florida State University(佛罗里达州立大学);CNRS(法国国家科学研究中心)
最近的研究表明,重尾可以出现在随机最优化和尾部的沉重有联系的泛化错误。虽然这些研究揭示了现代环境中泛化行为的有趣方面,但它们依赖于强拓扑和统计规律性假设,这些假设很难在实践中得到验证。此外,经验表明,重尾和泛化之间的关系在实践中可能并不总是单调的,与现有理论的结论相反。在这项研究中,我们建立了新的尾部行为和随机梯度下降(SGD)的泛化性能之间的联系,通过镜头的算法稳定性。我们考虑一个二次优化问题,并使用一个重尾随机微分方程(及其欧拉离散化)作为一个代理建模的重尾行为出现在SGD。然后,我们证明了一致的稳定性界限,这揭示了以下结果:(i)没有任何外来的假设,我们表明,SGD将不会是稳定的,如果稳定性是衡量平方损失$x\mapsto x^2$,而它反过来成为稳定的,如果稳定性,而不是衡量与一些$p<2$的替代损失$x\mapsto |x|^p$。(ii)根据数据的方差,存在\emph{`threshold of heavy-tailedness'},使得泛化误差随着尾部变重而减小,只要尾部比该阈值轻。这表明,重尾和泛化之间的关系不是全局单调的。(iii)我们证明了匹配的一致稳定的下界,这意味着我们的界限是紧的沉重的尾巴。我们支持我们的理论与合成和真实的神经网络实验。
Mean estimation and regression under heavy-tailed distributions--a survey
Pompeu Fabra University(庞培法布拉大学);ICREA(加泰罗尼亚研究与高等研究院);Barcelona Graduate School of Economics(巴塞罗那经济研究生院);The Australian National University(澳大利亚国立大学);Sorbonne University(索邦大学)