This article suggests that deterministic Gradient Descent, which does not use any stochastic gradient approximation, can still exhibit stochastic behaviors. In particular, it shows that if the objective function exhibit multiscale behaviors, then in a large learning rate regime which only resolves the macroscopic but not the microscopic details of the objective, the deterministic GD dynamics can become chaotic and convergent not to a local minimizer but to a statistical distribution. A sufficient condition is also established for approximating this long-time statistical limit by a rescaled Gibbs distribution. Both theoretical and numerical demonstrations are provided, and the theoretical part relies on the construction of a stochastic map that uses bounded noise (as opposed to discretized diffusions).
Data-induced multiscale losses and efficient multirate gradient descent schemes
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Oden Institute for Computational Engineering and Sciences（奥登计算工程与科学研究所）
Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
In this paper we investigate how gradient-based algorithms such as gradient descent, (multi-pass) stochastic gradient descent, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best generalization error at limited sample complexity. We consider the loss landscape of the high-dimensional phase retrieval problem as a prototypical highly non-convex example. We observe that for phase retrieval the stochastic variants of gradient descent are able to reach perfect generalization for regions of control parameters where the gr…
KAIST（韩国科学技术院）；Center for Superintelligence, SNU（首尔大学超级智能中心）；New York University（纽约大学）；Facebook AI Research（Facebook人工智能研究院）
Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is limited. In recent years, some researchers have used the stochasticity of minibatch gradients, or the signal-to-noise ratio, to better characterize the learning dynamics of SGD. Inspired from these work, we here analyze SGD from a geometrical perspective by inspecting the stochasticity of the norms and directions of minibatch gradients. We propose a model of the directional concentration for minibatch gradients thro…
Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dynamics of stochastic gradient descent over diagonal linear networks through its continuous time version, namely stochastic gradient flow. We explicitly characterise the solution chosen by the stochastic flow and prove that it always enjoys better generalisation properties than that of gradient flow. Quite surprisingly, we show that the convergence speed of the training loss controls the magnitude of the biasing effect…
University of California, Berkeley（加州大学伯克利分校）；Microsoft Research, India（微软研究院（印度））；Duke University（杜克大学）；University of Washington, Seattle（华盛顿大学西雅图分校）
Gradient descent (GD) and stochastic gradient descent (SGD) are the workhorses of large-scale machine learning. While classical theory focused on analyzing the performance of these methods in convex optimization problems, the most notable successes in machine learning have involved nonconvex optimization, and a gap has arisen between theory and practice. Indeed, traditional analyses of GD and SGD show that both algorithms converge to stationary points efficiently. But these analyses do not take into account the possibility of converging to saddle points. More recent theory has shown that GD an…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Nevada（内华达大学）
In this work, multiplicative stochasticity is applied to the learning rate of stochastic optimization algorithms, giving rise to stochastic learning-rate schemes. In-expectation theoretical convergence results of Stochastic Gradient Descent equipped with this novel stochastic learning rate scheme under the stochastic setting, as well as convergence results under the online optimization settings are provided. Empirical results consider the case of an adaptively uniformly distributed multiplicative stochasticity and include not only Stochastic Gradient Descent, but also other popular algorithms …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Nevada（内华达大学）
Large-scale optimization problems require algorithms both effective and efficient. One such popular and proven algorithm is Stochastic Gradient Descent which uses first-order gradient information to solve these problems. This paper studies almost-sure convergence rates of the Stochastic Gradient Descent method when instead of deterministic, its learning rate becomes stochastic. In particular, its learning rate is equipped with a multiplicative stochasticity, producing a stochastic learning rate scheme. Theoretical results show accelerated almost-sure convergence rates of Stochastic Gradient De…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Gradient descent is a popular algorithm in optimization, and its performance in convex settings is mostly well understood. In non-convex settings, it has been shown that gradient descent is able to escape saddle points asymptotically and converge to local minimizers [Lee et. al. 2016]. Recent studies also show a perturbed version of gradient descent is enough to escape saddle points efficiently [Jin et. al. 2015, Ge et. al. 2017]. In this paper we show a negative result: gradient descent may take exponential time to escape saddle points, with non-pathological two dimensional functions. While o…
Bocconi Institute for Data Science and Analytics, Bocconi University（博科尼大学博科尼数据科学与分析研究所）；Italian Institute for Genomic Medicine（意大利基因组医学研究所）；Istituto Nazionale di Fisica Nucleare（意大利国家核物理研究所）；Politecnico di Torino（都灵理工大学）；Radboud University Nijmegen（内梅亨大学）；Donders Institute for Brain, Cognition and Behaviour（唐德斯大脑、认知与行为研究所）；International Centre for Theoretical Physics（国际理论物理中心）
Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Binary solutions of the standard perceptron problem are obtained from a simple gradient descent procedure on a set of real values parametrizin…
University of Cambridge（剑桥大学）；Harvard University（哈佛大学）
Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the art results for many problems. Nonetheless, Stochastic Gradient Descent typically cannot find the global minimum, thus its empirical effectiveness is hitherto mysterious. We derive a correspondence between parameter inference and free energy minimisation in statistical physics. The degree of undersampling plays the role of temperature. Analogous to the energy-entropy competition in statistical physics, wide but shal…
Losing momentum in continuous-time stochastic optimisation
Princeton University（普林斯顿大学）；The University of Manchester（曼彻斯特大学）；Technische Universiteit Delft（代尔夫特理工大学）；Technische Universität München（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Recommender systems need to mirror the complexity of the environment they are applied in. The more we know about what might benefit the user, the more objectives the recommender system has. In addition there may be multiple stakeholders - sellers, buyers, shareholders - in addition to legal and ethical constraints. Simultaneously optimizing for a multitude of objectives, correlated and not correlated, having the same scale or not, has proven difficult so far. We introduce a stochastic multi-gradient descent approach to recommender systems (MGDRec) to solve this problem. We show that this excee…
Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to their success in practical problems, and there is some understanding of the role of stochastic gradient descent in this context. Momentum modifications of gradient descent such as Polyak's Heavy Ball method (HB) and Nesterov's method of accelerated gradients (NAG), are also widely adopted. In this work our focus is on understanding the role of momentum in t…
Stanford University（斯坦福大学）；Microsoft Research（微软研究院）；University of Washington（华盛顿大学）
Dropout is a widely-used regularization technique, often required to obtain state-of-the-art for a number of architectures. This work demonstrates that dropout introduces two distinct but entangled regularization effects: an explicit effect (also studied in prior work) which occurs since dropout modifies the expected training objective, and, perhaps surprisingly, an additional implicit effect from the stochasticity in the dropout training update. This implicit regularization effect is analogous to the effect of stochasticity in small mini-batch stochastic gradient descent. We disentangle these…
Byzantine-Robust Distributed SGD: A Unified Analysis and Tight Error Bounds
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学前沿交叉科学学院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）；Ministry of Education Key Laboratory of Mathematics and Information Networks, School of Mathematical Sciences, Beijing University of Posts and Telecommunications（北京邮电大学数学科学学院数学与信息网络教育部重点实验室）
Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；CEA（法国原子能和替代能源委员会）；Institut de physique théorique（理论物理研究所）
Stochastic Gradient Descent (SGD) is the workhorse algorithm of deep learning technology. At each step of the training phase, a mini batch of samples is drawn from the training dataset and the weights of the neural network are adjusted according to the performance on this specific subset of examples. The mini-batch sampling procedure introduces a stochastic dynamics to the gradient descent, with a non-trivial state-dependent noise. We characterize the stochasticity of SGD and a recently-introduced variant, \emph{persistent} SGD, in a prototypical neural network model. In the under-parametrized…
Stochastic noise can be helpful for variational quantum algorithms
Pritzker School of Molecular Engineering, The University of Chicago（普利兹克分子工程学院，芝加哥大学）；School of Computing and Information, University of Pittsburgh（计算与信息学院，匹兹堡大学）；Chicago Quantum Exchange（芝加哥量子交换）；Kadanoff Center for Theoretical Physics, The University of Chicago（卡达诺夫理论物理中心，芝加哥大学）；Dahlem Center for Complex Quantum Systems, Freie Universität Berlin（达赫勒复杂量子系统中心，柏林自由大学）；Electrical Engineering and Computer Science, Technische Universität Berlin（电气工程与计算机科学，柏林技术大学）；Helmholtz-Zentrum Berlin für Materialien und Energie（柏林材料与能源研究所）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）
A quadratic approximation of neural network loss landscapes has been extensively used to study the optimization process of these networks. Though, it usually holds in a very small neighborhood of the minimum, it cannot explain many phenomena observed during the optimization process. In this work, we study the structure of neural network loss functions and its implication on optimization in a region beyond the reach of a good quadratic approximation. Numerically, we observe that neural network loss functions possesses a multiscale structure, manifested in two ways: (1) in a neighborhood of mini…
This paper extends the deep material network (DMN) proposed by Liu et al. (2019) to tackle general 3-dimensional (3D) problems with arbitrary material and geometric nonlinearities. It discovers a new way of describing multiscale heterogeneous materials by a multi-layer network structure and mechanistic building blocks. The data-driven framework of DMN is discussed in detail about the offline training and online extrapolation stages. Analytical solutions of the 3D building block with a two-layer structure in both small- and finite-strain formulations are derived based on interfacial equilibrium…
Distributionally Time-Varying Online Stochastic Optimization under Polyak-Łojasiewicz Condition with Application in Conditional Value-at-Risk Statistical Learning
The Australian National University（澳大利亚国立大学）；The University of Melbourne（墨尔本大学）
Nordita（北欧理论物理研究所）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Stockholm University（斯德哥尔摩大学）；École Normale Supérieure, Université PSL（巴黎文理研究大学高等师范学院）；CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）
Recent studies have shown that gradient descent (GD) can achieve improved generalization when its dynamics exhibits a chaotic behavior. However, to obtain the desired effect, the step-size should be chosen sufficiently large, a task which is problem dependent and can be difficult in practice. In this study, we incorporate a chaotic component to GD in a controlled manner, and introduce multiscale perturbed GD (MPGD), a novel optimization framework where the GD recursion is augmented with chaotic perturbations that evolve via an independent dynamical system. We analyze MPGD from three different …
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；University of Ottawa（渥太华大学）；Ghent University（根特大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）
We examine the zero-temperature Metropolis Monte Carlo algorithm as a tool for training a neural network by minimizing a loss function. We find that, as expected on theoretical grounds and shown empirically by other authors, Metropolis Monte Carlo can train a neural net with an accuracy comparable to that of gradient descent, if not necessarily as quickly. The Metropolis algorithm does not fail automatically when the number of parameters of a neural network is large. It can fail when a neural network's structure or neuron activations are strongly heterogenous, and we introduce an adaptive Mont…
Bilevel Learning via Inexact Stochastic Gradient Descent
EPSRC Centre for Doctoral Training in Statistical Applied Mathematics at Bath（巴斯大学统计应用数学博士培训中心）；Australian Research Council（澳大利亚研究理事会）；Indian Institute of Technology (IIT), Kharagpur（印度理工学院（Kharagpur分校））；University of Melbourne（墨尔本大学）；Department of Electronics & Electrical Communication Engineering（电子与电气通信工程系）；School of Mathematics and Statistics（数学与统计学学院）
Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；University of Washington（华盛顿大学）
We describe a graph-based semi-supervised learning framework in the context of deep neural networks that uses a graph-based entropic regularizer to favor smooth solutions over a graph induced by the data. The main contribution of this work is a computationally efficient, stochastic graph-regularization technique that uses mini-batches that are consistent with the graph structure, but also provides enough stochasticity (in terms of mini-batch data diversity) for convergence of stochastic gradient descent methods to good solutions. For this work, we focus on results of frame-level phone classifi…
Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；University of Washington（华盛顿大学）
We describe a computationally efficient, stochastic graph-regularization technique that can be utilized for the semi-supervised training of deep neural networks in a parallel or distributed setting. We utilize a technique, first described in [13] for the construction of mini-batches for stochastic gradient descent (SGD) based on synthesized partitions of an affinity graph that are consistent with the graph structure, but also preserve enough stochasticity for convergence of SGD to good local minima. We show how our technique allows a graph-based semi-supervised loss function to be decomposed i…
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Training machine learning (ML) models typically involves expensive iterative optimization. Once the model's final parameters are released, there is currently no mechanism for the entity which trained the model to prove that these parameters were indeed the result of this optimization procedure. Such a mechanism would support security of ML applications in several ways. For instance, it would simplify ownership resolution when multiple parties contest ownership of a specific model. It would also facilitate the distributed training across untrusted workers where Byzantine workers might otherwise…
University of California, Berkeley（加州大学伯克利分校）；The Ohio State University（俄亥俄州立大学）
Stochastic differential equations (SDEs) are used to describe a wide variety of complex stochastic dynamical systems. Learning the hidden physics within SDEs is crucial for unraveling fundamental understanding of these systems' stochastic and nonlinear behavior. We propose a flexible and scalable framework for training artificial neural networks to learn constitutive equations that represent hidden physics within SDEs. The proposed stochastic physics-informed neural ordinary differential equation framework (SPINODE) propagates stochasticity through the known structure of the SDE (i.e., the kno…
Recent analyses of certain gradient descent optimization methods have shown that performance can degrade in some settings - such as with stochasticity or implicit momentum. In deep reinforcement learning (Deep RL), such optimization methods are often used for training neural networks via the temporal difference error or policy gradient. As an agent improves over time, the optimization target changes and thus the loss landscape (and local optima) change. Due to the failure modes of those methods, the ideal choice of optimizer for Deep RL remains unclear. As such, we provide an empirical analysi…
Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent
Laboratory for Big Data and Decision（大数据与决策实验室）；National University of Defense Technology（国防科技大学）；Samsung AI Center Cambridge（三星AI研究中心）；Queen Mary University of London（伦敦玛丽女王大学）；Fudan University（复旦大学）；ShanghaiTech University（上海科技大学）；Shanghai Innovation Institute（上海创新研究院）
Peng Cheng Laboratory（鹏城实验室）；Fudan University（复旦大学）；Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）；Politecnico di Milano（米兰理工大学）
