High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Institute of Control Sciences RAS（俄罗斯科学院控制科学研究所）；National Research University Higher School of Economics（国家研究型大学高等经济学院）；Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机学研究所）；Innopolis University（因诺波利斯大学）；Institute for Information Transmission Problems RAS（俄罗斯科学院信息传输问题研究所）
Eduard Gorbunov, Marina Danilova, Innokentiy Shibaev, Pavel Dvurechensky, Alexander Gasnikov
备注：
61 pages, 12 figures. Changes in V2: different presentation of the results, different structure, new experiments. Changes in V3: some typos were fixed
Stochastic first-order methods are standard for training large-scale machine learning models. Random behavior may cause a particular run of an algorithm to result in a highly suboptimal objective value, whereas theoretical guarantees are usually proved for the expectation of the objective value. Thus, it is essential to theoretically guarantee that algorithms provide small objective residual with high probability. Existing methods for non-smooth stochastic convex optimization have complexity bounds with the dependence on the confidence level that is either negative-power or logarithmic but under an additional assumption of sub-Gaussian (light-tailed) noise distribution that may not hold in practice. In our paper, we resolve this issue and derive the first high-probability convergence results with logarithmic dependence on the confidence level for non-smooth convex stochastic optimization problems with non-sub-Gaussian (heavy-tailed) noise. To derive our results, we propose novel stepsize rules for two stochastic methods with gradient clipping. Moreover, our analysis works for generalized smooth objectives with Hölder-continuous gradients, and for both methods, we provide an extension for strongly convex problems. Finally, our results imply that the first (accelerated) method we consider also has optimal iteration and oracle complexity in all the regimes, and the second one is optimal in the non-smooth setting.
Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
KAUST Center of Excellence for Generative AI（知识增强人工智能卓越中心）；ETH Zurich（苏黎世联邦理工学院）；ETH AI Center（苏黎世联邦理工学院人工智能中心）；Georgia Institute of Technology（佐治亚理工学院）
Accelerated stochastic first-order method for convex optimization under heavy-tailed noise
Department of Mathematics, Linköping University, Sweden（林雪平大学数学系）；Department of Industrial and Systems Engineering, University of Minnesota, USA（明尼苏达大学工业与系统工程系）
Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clipping
Inria, CNRS, Ecole Normale Supérieure PSL Research University（法国国家科学研究中心、巴黎高等师范学院PSL研究大学）；Department of Management Science and Information Systems（管理科学与信息系统系）；Faculty of Engineering and Natural Sciences（工程与自然科学学院）；Department of Mathematics（数学系）
