Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Alan Turing Institute（艾伦·图灵研究所）；University of Oxford（牛津大学）；Florida State University（佛罗里达州立大学）；Rutgers Business School（罗格斯商学院）；INRIA（法国国家信息与自动化研究所）；École Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）
Alexander Camuto, Xiaoyu Wang, Lingjiong Zhu, Chris Holmes, Mert Gürbüzbalaban, Umut Şimşekli
Gaussian noise injections (GNIs) are a family of simple and widely-used regularisation methods for training neural networks, where one injects additive or multiplicative Gaussian noise to the network activations at every iteration of the optimisation algorithm, which is typically chosen as stochastic gradient descent (SGD). In this paper we focus on the so-called `implicit effect' of GNIs, which is the effect of the injected noise on the dynamics of SGD. We show that this effect induces an asymmetric heavy-tailed noise on SGD gradient updates. In order to model this modified dynamics, we first develop a Langevin-like stochastic differential equation that is driven by a general family of asymmetric heavy-tailed noise. Using this model we then formally prove that GNIs induce an `implicit bias', which varies depending on the heaviness of the tails and the level of asymmetry. Our empirical results confirm that different types of neural networks trained with GNIs are well-modelled by the proposed dynamics and that the implicit effect of these injections induces a bias that degrades the performance of networks.
Implicitly normalized forecaster with clipping for linear and non-linear heavy-tailed multi-armed bandits
MSU Institute for Artificial Intelligence（莫斯科国立大学人工智能研究所）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Institute for Information Transmission Problems（信息传输问题研究所）；V.A. Trapeznikov Institute of Control Sciences of Russian Academy of Sciences（俄罗斯科学院特拉佩兹尼科夫控制科学研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Skoltech（斯科尔科沃科学技术学院）；ISP RAS Research Center for Trusted Artificial Intelligence（俄罗斯科学院系统编程研究所可信人工智能研究中心）
Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD
Paris Research Center, Huawei Technologies France（华为技术法国巴黎研发中心）；Boğaziçi University（博加齐大学）；Université Gustave Eiffel（古斯塔夫·埃菲尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）
University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper shows that the implicit bias of gradient descent on linearly separable data is exactly characterized by the optimal solution of a dual optimization problem given by a smoothed margin, even for general losses. This is in contrast to prior results, which are often tailored to exponentially-tailed losses. For the exponential loss specifically, with $n$ training examples and $t$ gradient descent steps, our dual analysis further allows us to prove an $O(\ln(n)/\ln(t))$ convergence rate to the $\ell_2$ maximum margin direction, when a constant step size is used. This rate is tight in both…
In this paper, we leverage over-parameterization to design regularization-free algorithms for the high-dimensional single index model and provide theoretical guarantees for the induced implicit regularization phenomenon. Specifically, we study both vector and matrix single index models where the link function is nonlinear and unknown, the signal parameter is either a sparse vector or a low-rank symmetric matrix, and the response variable can be heavy-tailed. To gain a better understanding of the role played by implicit regularization without excess technicality, we assume that the distribution…
Kernel methods have achieved very good performance on large scale regression and classification problems, by using the Nyström method and preconditioning techniques. The Nyström approximation -- based on a subset of landmarks -- gives a low rank approximation of the kernel matrix, and is known to provide a form of implicit regularization. We further elaborate on the impact of sampling diverse landmarks for constructing the Nyström approximation in supervised as well as unsupervised kernel methods. By using Determinantal Point Processes for sampling, we obtain additional theoretical results con…
Tel Aviv University（特拉维夫大学）；Google Research（谷歌研究院）
An influential line of recent work has focused on the generalization properties of unregularized gradient-based learning procedures applied to separable linear classification with exponentially-tailed loss functions. The ability of such methods to generalize well has been attributed to the their implicit bias towards large margin predictors, both asymptotically as well as in finite time. We give an additional unified explanation for this generalization and relate it to two simple properties of the optimization objective, that we refer to as realizability and self-boundedness. We introduce a ge…
Fast Computation of Superquantile-Constrained Optimization Through Implicit Scenario Reduction
Department of Industrial and Systems Engineering, University of Minnesota（明尼苏达大学工业与系统工程系）；Department of Industrial Engineering and Operations Research, University of California, Berkeley（加州大学伯克利分校工业工程与运筹学系）
Calculation Consulting（计算咨询公司）；ICSI（国际计算机科学研究所）；University of California at Berkeley（加州大学伯克利分校）
Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that the empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of regularization, such as Dropout or Weight Norm constraints. Building on recent res…
Calculation Consulting（Calculation Consulting公司）；ICSI（国际计算机科学研究所）；University of California at Berkeley（加州大学伯克利分校）
Random Matrix Theory (RMT) is applied to analyze weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that the DNN training process itself implicitly implements a form of Self-Regularization. The empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of e…
Training More Robust Classification Model via Discriminative Loss and Gaussian Noise Injection
Ampere Software Technology（Ampere软件技术公司）；Institut de mathématiques de Toulouse（图卢兹数学研究所）；Institut de Recherche en Informatique de Toulouse（图卢兹计算机研究所）；Laboratoire Jean Alexandre Dieudonné, Université Côte d’Azur（坎特伯雷大学Dieudonné实验室）
Nordita, KTH Royal Institute of Technology（北欧理论物理研究所，瑞典皇家理工学院）；Stockholm University（斯德哥尔摩大学）；University of Pittsburgh（匹兹堡大学）；ICSI（国际计算机科学研究所）；UC Berkeley（加州大学伯克利分校）
We provide a general framework for studying recurrent neural networks (RNNs) trained by injecting noise into hidden states. Specifically, we consider RNNs that can be viewed as discretizations of stochastic differential equations driven by input data. This framework allows us to study the implicit regularization effect of general noise injection schemes by deriving an approximate explicit regularizer in the small noise regime. We find that, under reasonable assumptions, this implicit regularization promotes flatter minima; it biases towards models with more stable dynamics; and, in classificat…
