We study scalable alternatives to robust gradient descent (RGD) techniques that can be used when the losses and/or gradients can be heavy-tailed, though this will be unknown to the learner. The core technique is simple: instead of trying to robustly aggregate gradients at each step, which is costly and leads to sub-optimal dimension dependence in risk bounds, we instead focus computational effort on robustly choosing (or newly constructing) a strong candidate based on a collection of cheap stochastic sub-processes which can be run in parallel. The exact selection process depends on the convexity of the underlying objective, but in all cases, our selection technique amounts to a robust form of boosting the confidence of weak learners. In addition to formal guarantees, we also provide empirical analysis of robustness to perturbations to experimental conditions, under both sub-Gaussian and heavy-tailed data, along with applications to a variety of benchmark datasets. The overall take-away is an extensible procedure that is simple to implement, trivial to parallelize, which keeps the formal merits of RGD methods but scales much better to large learning problems.
Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetry
Carnegie Mellon University（卡内基梅隆大学）；Faculty of Technical Sciences, University of Novi Sad（诺维萨德大学技术科学学院）；Faculty of Sciences, University of Novi Sad（诺维萨德大学理学院）
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎电信学院，巴黎理工学院）；Université Paris Cité, Université Paris Saclay, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli（巴黎西岱大学，巴黎萨克雷大学，巴黎高等师范学校巴黎萨克雷分校，法国国家科学研究中心，SSA，INSERM，博尔莱中心）
Heavy-Tailed Diffusion with Denoising Lévy Probabilistic Models
INRIA - Department of Computer Science, PSL Research University, Paris, France（INRIA-计算机科学系，PSL研究大学，法国巴黎）；École Polytechnique - CMAP, IP Paris, Palaiseau, France（巴黎综合理工学院-CMAP，IP巴黎，法国帕莱苏）
Efficient Group Lasso Regularized Rank Regression with Simulation-Based Tuning
Engineering Systems and Design, Singapore University of Technology and Design, Singapore（新加坡科技设计大学工程系统与设计系）；School of Mathematics and Statistics, Henan University（河南大学数学与统计学院）；School of Mathematics and Statistics & Center for Applied Mathematics of Henan Province, Henan University（河南大学数学与统计学院及河南省应用数学中心）
