In this paper, we investigate the limiting behavior of a continuous-time counterpart of the Stochastic Gradient Descent (SGD) algorithm applied to two-layer overparameterized neural networks, as the number or neurons (ie, the size of the hidden layer) $N \to +\infty$. Following a probabilistic approach, we show 'propagation of chaos' for the particle system defined by this continuous-time dynamics under different scenarios, indicating that the statistical interaction between the particles asymptotically vanishes. In particular, we establish quantitative convergence with respect to $N$ of any particle to a solution of a mean-field McKean-Vlasov equation in the metric space endowed with the Wasserstein distance. In comparison to previous works on the subject, we consider settings in which the sequence of stepsizes in SGD can potentially depend on the number of neurons and the iterations. We then identify two regimes under which different mean-field limits are obtained, one of them corresponding to an implicitly regularized version of the minimization problem at hand. We perform various experiments on real datasets to validate our theoretical results, assessing the existence of these two regimes on classification problems and illustrating our convergence results.
Parameter Estimation in Stochastic Differential Equations via Wiener Chaos Expansion and Stochastic Gradient Descent
Universitat Autónoma de Barcelona, Departament de Matemàtiques（巴塞罗那自治大学数学系）；Facultad de Ciencias - Universidad Nacional Autónoma de México（墨西哥国立自治大学理学院）；Secihti - Centro Interdisciplinario de Ciencias Marinas. IPN.（国家理工学院海洋科学跨学科中心）
Error estimates between SGD with momentum and underdamped Langevin diffusion
Laboratoire de Mathematiques Blaise Pascal, CNRS-UMR 6620, Universite Clermont-Auvergne（克莱蒙奥弗涅大学）；University of Macau（澳门大学）；UM Zhuhai Research Institute（澳门大学珠海研究院）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Beijing International Center for Mathematical Research (BICMR), Peking University（北京大学北京国际数学研究中心）
Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD
Paris Research Center, Huawei Technologies France（华为技术法国巴黎研发中心）；Boğaziçi University（博加齐大学）；Université Gustave Eiffel（古斯塔夫·埃菲尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）
We develop a stochastic differential equation, called homogenized SGD, for analyzing the dynamics of stochastic gradient descent (SGD) on a high-dimensional random least squares problem with $\ell^2$-regularization. We show that homogenized SGD is the high-dimensional equivalence of SGD -- for any quadratic statistic (e.g., population risk with quadratic loss), the statistic under the iterates of SGD converges to the statistic under homogenized SGD when the number of samples $n$ and number of features $d$ are polynomially related ($d^c < n < d^{1/c}$ for some $c > 0$). By analyzing homogenized…
Optimization in Deep Learning is mainly guided by vague intuitions and strong assumptions, with a limited understanding how and why these work in practice. To shed more light on this, our work provides some deeper understandings of how SGD behaves by empirically analyzing the trajectory taken by SGD from a line search perspective. Specifically, a costly quantitative analysis of the full-batch loss along SGD trajectories from common used models trained on a subset of CIFAR-10 is performed. Our core results include that the full-batch loss along lines in update step direction is highly parabolic…
Mini-batch SGD with momentum is a fundamental algorithm for learning large predictive models. In this paper we develop a new analytic framework to analyze noise-averaged properties of mini-batch SGD for linear models at constant learning rates, momenta and sizes of batches. Our key idea is to consider the dynamics of the second moments of model parameters for a special family of "Spectrally Expressible" approximations. This allows to obtain an explicit expression for the generating function of the sequence of loss values. By analyzing this generating function, we find, in particular, that 1) t…
The University of Tokyo（东京大学）；RIKEN Center for AIP（理化学研究所人工智能中心）
Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum that often generalizes well. However, it is mathematically unclear how deep learning can select a flat minimum among so many minima. To answer the question quantitatively, we develop a density diffusion theory (DDT) to reveal how minima selection quantitatively depends on the minima sharpness and the hyperparameters. To the best of our knowledge, we are the first to theoretically and empirically prove that, benefited from the Hessian-dependent co…
Stochastic gradient algorithms are often unstable when applied to functions that do not have Lipschitz-continuous and/or bounded gradients. Gradient clipping is a simple and effective technique to stabilize the training process for problems that are prone to the exploding gradient problem. Despite its widespread popularity, the convergence properties of the gradient clipping heuristic are poorly understood, especially for stochastic problems. This paper establishes both qualitative and quantitative convergence results of the clipped stochastic (sub)gradient method (SGD) for non-smooth convex f…
Is Stochastic Gradient Descent Effective? A PDE Perspective on Machine Learning processes
Departamento de Matemáticas, Universidad Autónoma de Madrid, ICMAT - Instituto de Ciencias Matemáticas, CSIC-UAM-UC3M-UCM（数学系，马德里自治大学，ICMAT数学科学研究所，CSIC-UAM-UC3M-UCM）
Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent
Mathematical Institute for Machine Learning and Data Science, Catholic University of Eichstätt-Ingolstadt（机器学习与数据科学数学研究所，埃施特哈特-因戈尔施塔特天主教大学）；Department of Computer Science, RPTU Kaiserslautern-Landau（计算机科学系，凯斯莱特恩-兰道大学）；Department of Mathematics, The University of Hong Kong（数学系，香港大学）；School of Mathematics and Statistics, The University of Sydney（数学与统计学学院，悉尼大学）
It has been empirically observed that the flatness of minima obtained from training deep networks seems to correlate with better generalization. However, for deep networks with positively homogeneous activations, most measures of sharpness/flatness are not invariant to rescaling of the network parameters, corresponding to the same function. This means that the measure of flatness/sharpness can be made as small or as large as possible through rescaling, rendering the quantitative measures meaningless. In this paper we show that for deep networks with positively homogenous activations, these res…
National Institute of Science Education and Research, Bhubaneswar（布巴内斯瓦尔国家科学教育与研究学院）；Homi Bhabha National Institute（霍米·巴巴国家研究所）；Max Planck Institute for Gravitational Physics (Albert Einstein Institute)（马克斯·普朗克引力物理研究所（阿尔伯特·爱因斯坦研究所））；Jadavpur University（贾达普大学）；Mississippi State University（密西西比州立大学）
In this work, our prime objective is to study the phenomena of quantum chaos and complexity in the machine learning dynamics of Quantum Neural Network (QNN). A Parameterized Quantum Circuits (PQCs) in the hybrid quantum-classical framework is introduced as a universal function approximator to perform optimization with Stochastic Gradient Descent (SGD). We employ a statistical and differential geometric approach to study the learning theory of QNN. The evolution of parametrized unitary operators is correlated with the trajectory of parameters in the Diffusion metric. We establish the parametriz…
Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems
Sapienza Università di Roma（罗马第一大学）；Istituto Nazionale di Fisica Nucleare（国家核物理研究所）；Osaka University（大阪大学）；Università degli studi di Firenze（佛罗伦萨大学）；Institute of Nanotechnology (NANOTEC) - CNR（国家研究委员会纳米技术研究所）
本文证明在离散优化和推断问题中，类 SGD 算法动力学与 Metropolis Monte Carlo 动力学定量匹配，并利用此等价性优化 mini-batch 大小以提升困难推断问题中的信号恢复效率。
We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results reveal an order-to-chaos expressivity phase transition, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth but not width. We prove this generic class of deep random functions cannot be efficiently computed by any shallow network, going beyond prior work restricted to the analysis of single functions. Moreover, we formalize and quantitativ…
