Gradient clipping is a commonly used technique to stabilize the training process of neural networks. A growing body of studies has shown that gradient clipping is a promising technique for dealing with the heavy-tailed behavior that emerged in stochastic optimization as well. While gradient clipping is significant, its theoretical guarantees are scarce. Most theoretical guarantees only provide an in-expectation analysis and only focus on optimization performance. In this paper, we provide high probability analysis in the non-convex setting and derive the optimization bound and the generalization bound simultaneously for popular stochastic optimization algorithms with gradient clipping, including stochastic gradient descent and its variants of momentum and adaptive stepsizes. With the gradient clipping, we study a heavy-tailed assumption that the gradients only have bounded $α$-th moments for some $α\in (1, 2]$, which is much weaker than the standard bounded second-moment assumption. Overall, our study provides a relatively complete picture for the theoretical guarantee of stochastic optimization algorithms with clipping.
Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
KAUST Center of Excellence for Generative AI（知识增强人工智能卓越中心）；ETH Zurich（苏黎世联邦理工学院）；ETH AI Center（苏黎世联邦理工学院人工智能中心）；Georgia Institute of Technology（佐治亚理工学院）
Accelerated stochastic first-order method for convex optimization under heavy-tailed noise
Department of Mathematics, Linköping University, Sweden（林雪平大学数学系）；Department of Industrial and Systems Engineering, University of Minnesota, USA（明尼苏达大学工业与系统工程系）
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
Moscow Institute of Physics and Technology, Russia（莫斯科物理技术学院）；Ivannikov Institute for System Programming RAS, Russia（伊万诺夫系统编程研究所）；Sber AI Lab, Russia（俄罗斯Sber人工智能实验室）；Innopolis University, Russia（因诺波利斯大学）；The Russian Presidential Academy of National Economy and Public Administration, Russia（俄罗斯总统经济与公共行政学院）；Skolkovo Institute of Science and Technology, Russia（斯克沃尔佐夫科学与技术研究所）；Mohamed bin Zayed University of Artificial Intelligence, UAE（穆罕默德·本·扎耶德人工智能大学）；Machine Learning and Optimization Laboratory (MLO), EPFL, Lausanne, Switzerland（机器学习与优化实验室（MLO），EPFL，拉瓦尔普勒斯，瑞士）
Revisiting Gradient Normalization and Clipping for Nonconvex SGD under Heavy-Tailed Noise: Necessity, Sufficiency, and Acceleration
College of Computer Science and Technology National University of Defense Technology（计算机科学与技术学院 国防科技大学）；Center for Machine Learning Research (CMLR) Peking University（机器学习研究中心（CMLR） 北京大学）
Convergence Analysis of Over-the-Air FL with Compression and Power Control via Clipping
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；King’s College London（伦敦国王学院）
High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noise
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）；Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机过程研究所）；Ivannikov Institute for System Programming RAS（伊万尼科夫系统编程研究所（俄罗斯科学院））；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）
Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise
Department of Mathematics, University of California, Berkeley, CA, USA（加州大学伯克利分校数学系）；Department of EECS, University of California, Berkeley, CA, USA（加州大学伯克利分校电子工程与计算机科学系）；Department of IEOR, University of California, Berkeley, CA, USA（加州大学伯克利分校工业工程与运营研究系）
随机一阶方法，如随机外梯度（SEG）或随机梯度下降上升（SGDA）解决光滑极大极小问题，更一般地说，变分不等式问题（VIP）已经获得了很多关注，近年来由于对抗公式在机器学习中的日益普及。然而，虽然已知高概率收敛界更准确地反映了随机方法的实际行为，但大多数收敛结果都是在预期中提供的。此外，唯一已知的高概率复杂性结果是在限制性亚高斯（轻尾）噪声和有界域假设下推导出来的[Juditsky et al.，2011年]。在这项工作中，我们证明了第一个高概率的复杂性结果与对数依赖的置信水平的随机方法解决单调和结构化非单调VIP与非次高斯（重尾）噪声和无界域。在单调的情况下，我们的结果与光尾情况下最著名的结果相匹配[Juditsky等人，2011]，并且对于结构化非单调问题（诸如负共单调、拟强单调和/或星共单调问题）是新颖的。我们通过研究SEG和SGDA与裁剪实现了这些结果。此外，我们数值验证了许多实际的GAN公式的梯度噪声是重尾的，并表明裁剪提高了SEG/SGDA的性能。
Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees
Carnegie Mellon University（卡内基梅隆大学）；Faculty of Technical Sciences, University of Novi Sad（诺威萨大学技术科学学院）；Faculty of Sciences, University of Novi Sad（诺威萨大学理学院）
Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clipping
Inria, CNRS, Ecole Normale Supérieure PSL Research University（法国国家科学研究中心、巴黎高等师范学院PSL研究大学）；Department of Management Science and Information Systems（管理科学与信息系统系）；Faculty of Engineering and Natural Sciences（工程与自然科学学院）；Department of Mathematics（数学系）
Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight
School of Computer Science and Big Data, Minjiang University（闽江学院计算机科学与大数据学院）；School of Information, Renmin University of China（中国人民大学信息学院）；RMIT University（皇家墨尔本理工大学）
DP-SGD-Global-Adapt-V2-S: Triad Improvements of Privacy, Accuracy and Fairness via Step Decay Noise Multiplier and Step Decay Upper Clipping Threshold
University of Louisiana at Lafayette（路易斯安那大学拉斐特分校）；College of Saint Benedict and Saint John’s University（圣本笃学院与圣约翰大学）；Mississippi State University（密西西比州立大学）
