We study the Stochastic Gradient Langevin Dynamics (SGLD) algorithm for non-convex optimization. The algorithm performs stochastic gradient descent, where in each step it injects appropriately scaled Gaussian noise to the update. We analyze the algorithm's hitting time to an arbitrary subset of the parameter space. Two results follow from our general theory: First, we prove that for empirical risk minimization, if the empirical risk is point-wise close to the (smooth) population risk, then the algorithm achieves an approximate local minimum of the population risk in polynomial time, escaping suboptimal local minima that only exist in the empirical risk. Second, we show that SGLD improves on one of the best known learnability results for learning linear classifiers under the zero-one loss.
We present a unified framework to analyze the global convergence of Langevin dynamics based algorithms for nonconvex finite-sum optimization with $n$ component functions. At the core of our analysis is a direct analysis of the ergodicity of the numerical approximations to Langevin dynamics, which leads to faster convergence rates. Specifically, we show that gradient Langevin dynamics (GLD) and stochastic gradient Langevin dynamics (SGLD) converge to the almost minimizer within $\tilde O\big(nd/(λε) \big)$ and $\tilde O\big(d^7/(λ^5ε^5) \big)$ stochastic gradient evaluations respectively, where…
Stochastic Gradient Descent Langevin Dynamics (SGLD) algorithms, which add noise to the classic gradient descent, are known to improve the training of neural networks in some cases where the neural network is very deep. In this paper we study the possibilities of training acceleration for the numerical resolution of stochastic control problems through gradient descent, where the control is parametrized by a neural network. If the control is applied at many discretization times then solving the stochastic control problem reduces to minimizing the loss of a very deep neural network. We numerical…
We prove quantitative convergence rates at which discrete Langevin-like processes converge to the invariant distribution of a related stochastic differential equation. We study the setup where the additive noise can be non-Gaussian and state-dependent and the potential function can be non-convex. We show that the key properties of these processes depend on the potential function and the second moment of the additive noise. We apply our theoretical findings to studying the convergence of Stochastic Gradient Descent (SGD) for non-convex problems and corroborate them with experiments using SGD to…
The stochastic gradient Langevin Dynamics is one of the most fundamental algorithms to solve sampling problems and non-convex optimization appearing in several machine learning applications. Especially, its variance reduced versions have nowadays gained particular attention. In this paper, we study two variants of this kind, namely, the Stochastic Variance Reduced Gradient Langevin Dynamics and the Stochastic Recursive Gradient Langevin Dynamics. We prove their convergence to the objective distribution in terms of KL-divergence under the sole assumptions of smoothness and Log-Sobolev inequalit…
Alfréd Rényi Institute of Mathematics（阿尔弗雷德·雷尼数学研究所）；The University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）；National Technical University of Athens（雅典国立技术大学）
Artificial neural networks (ANNs) are typically highly nonlinear systems which are finely tuned via the optimization of their associated, non-convex loss functions. In many cases, the gradient of any such loss function has superlinear growth, making the use of the widely-accepted (stochastic) gradient descent methods, which are based on Euler numerical schemes, problematic. We offer a new learning algorithm based on an appropriately constructed variant of the popular stochastic gradient Langevin dynamics (SGLD), which is called tamed unadjusted stochastic Langevin algorithm (TUSLA). We also pr…
Cornell University（康奈尔大学）；University of Connecticut（康涅狄格大学）
It is well known that adding any skew symmetric matrix to the gradient of Langevin dynamics algorithm results in a non-reversible diffusion with improved convergence rate. This paper presents a gradient algorithm to adaptively optimize the choice of the skew symmetric matrix. The resulting algorithm involves a non-reversible diffusion algorithm cross coupled with a stochastic gradient algorithm that adapts the skew symmetric matrix. The algorithm uses the same data as the classical Langevin algorithm. A weak convergence proof is given for the optimality of the choice of the skew symmetric matr…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；New York University（纽约大学）；Cornell University（康奈尔大学）
While low-precision optimization has been widely used to accelerate deep learning, low-precision sampling remains largely unexplored. As a consequence, sampling is simply infeasible in many large-scale scenarios, despite providing remarkable benefits to generalization and uncertainty estimation for neural networks. In this paper, we provide the first study of low-precision Stochastic Gradient Langevin Dynamics (SGLD), showing that its costs can be significantly reduced without sacrificing performance, due to its intrinsic ability to handle system noise. We prove that the convergence of low-pre…
Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction
University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；Kyushu Institute of Technology（九州工业大学）
Iowa State University（爱荷华州立大学）；New York University（纽约大学）
Deep generative models have emerged as a powerful class of priors for signals in various inverse problems such as compressed sensing, phase retrieval and super-resolution. Here, we assume an unknown signal to lie in the range of some pre-trained generative model. A popular approach for signal recovery is via gradient descent in the low-dimensional latent space. While gradient descent has achieved good empirical performance, its theoretical behavior is not well understood. In this paper, we introduce the use of stochastic gradient Langevin dynamics (SGLD) for compressed sensing with a generativ…
Langevin algorithms are gradient descent methods with additive noise. They have been used for decades in Markov chain Monte Carlo (MCMC) sampling, optimization, and learning. Their convergence properties for unconstrained non-convex optimization and learning problems have been studied widely in the last few years. Other work has examined projected Langevin algorithms for sampling from log-concave distributions restricted to convex compact sets. For learning and optimization, log-concave distributions correspond to convex losses. In this paper, we analyze the case of non-convex losses with comp…
On Stationary-Point Hitting Time and Ergodicity of Stochastic Gradient Langevin Dynamics
New York University（纽约大学）；Institute for Advanced Study（高等研究院）；National University of Singapore（新加坡国立大学）
随机梯度朗之万动力学（SGLD）是随机最优化中的一个基本算法。Zhang等人最近的工作。[2017]提出了一阶和二阶平稳点的SGLD命中时间的分析。Zhang et al. [2017]中的证明是一个通过界定Cheeger常数的两阶段过程，这是相当复杂的，并导致松散的界限。本文利用随机微分方程的直觉，直接分析了SGLD到达一阶和二阶平稳点的击中时间。我们的分析很简单。它只依赖于基本的线性代数和概率论工具。与Zhang等人[2017]相比，我们的直接分析也导致了更严格的界限，并显示了命中时间对不同因素的显式依赖性，包括维数，平滑度，噪声强度和步长效应。在适当的条件下，我们证明了SGLD到一阶稳定点的击中时间可以是维数无关的。此外，我们应用我们的分析研究机器学习中的几个重要的在线估计问题，包括线性回归，矩阵分解，和在线PCA。
Dimension-free convergence rates for gradient Langevin dynamics in RKHS
CREST, ENSAE, IP Paris（巴黎理工学院国立统计与经济管理学院经济与统计研究中心）；The University of Tokyo（东京大学）；INRIA, Université Paris-Saclay（巴黎萨克雷大学法国国家信息与自动化研究所）；Riken AIP（理化学研究所先进智能项目中心）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
We provide a new convergence analysis of stochastic gradient Langevin dynamics (SGLD) for sampling from a class of distributions that can be non-log-concave. At the core of our approach is a novel conductance analysis of SGLD using an auxiliary time-reversible Markov Chain. Under certain conditions on the target distribution, we prove that $\tilde O(d^4ε^{-2})$ stochastic gradient evaluations suffice to guarantee $ε$-sampling error in terms of the total variation distance, where $d$ is the problem dimension. This improves existing results on the convergence rate of SGLD (Raginsky et al., 2017;…
Nanjing University（南京大学）；the University of Iowa（爱荷华大学）
In this paper, we revisit the problem of smoothed online learning, in which the online learner suffers both a hitting cost and a switching cost, and target two performance metrics: competitive ratio and dynamic regret with switching cost. To bound the competitive ratio, we assume the hitting cost is known to the learner in each round, and investigate the simple idea of balancing the two costs by an optimization problem. Surprisingly, we find that minimizing the hitting cost alone is $\max(1, \frac{2}α)$-competitive for $α$-polyhedral functions and $1 + \frac{4}λ$-competitive for $λ$-quadratic …
What is the information leakage of an iterative randomized learning algorithm about its training data, when the internal state of the algorithm is \emph{private}? How much is the contribution of each specific training epoch to the information leakage through the released model? We study this problem for noisy gradient descent algorithms, and model the \emph{dynamics} of Rényi differential privacy loss throughout the training process. Our analysis traces a provably \emph{tight} bound on the Rényi divergence between the pair of probability distributions over parameters of models trained on neigh…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Minnesota, Twin Cities（明尼苏达大学双城分校）
Recent years have seen advances in generalization bounds for noisy stochastic algorithms, especially stochastic gradient Langevin dynamics (SGLD) based on stability (Mou et al., 2018; Li et al., 2020) and information theoretic approaches (Xu and Raginsky, 2017; Negrea et al., 2019; Steinke and Zakynthinou, 2020). In this paper, we unify and substantially generalize stability based generalization bounds and make three technical contributions. First, we bound the generalization error in terms of expected (not uniform) stability which arguably leads to quantitatively sharper bounds. Second, as ou…
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院LTCI实验室）；Department of Statistics, University of Oxford（牛津大学统计系）；Department of Mathematics, Florida State University（佛罗里达州立大学数学系）；Department of Management Science and Information Systems, Rutgers Business School（罗格斯大学商学院管理科学与信息系统系）
Stochastic gradient descent with momentum (SGDm) is one of the most popular optimization algorithms in deep learning. While there is a rich theory of SGDm for convex problems, the theory is considerably less developed in the context of deep learning where the problem is non-convex and the gradient noise might exhibit a heavy-tailed behavior, as empirically observed in recent studies. In this study, we consider a \emph{continuous-time} variant of SGDm, known as the underdamped Langevin dynamics (ULD), and investigate its asymptotic properties under heavy-tailed perturbations. Supported by recen…
In this work, we unify several expected generalization error bounds based on random subsets using the framework developed by Hellström and Durisi [1]. First, we recover the bounds based on the individual sample mutual information from Bu et al. [2] and on a random subset of the dataset from Negrea et al. [3]. Then, we introduce their new, analogous bounds in the randomized subsample setting from Steinke and Zakynthinou [4], and we identify some limitations of the framework. Finally, we extend the bounds from Haghifam et al. [5] for Langevin dynamics to stochastic gradient Langevin dynamics and…
We consider the problem of sampling from a target distribution, which is \emph {not necessarily logconcave}, in the context of empirical risk minimization and stochastic optimization as presented in Raginsky et al. (2017). Non-asymptotic analysis results are established in the $L^1$-Wasserstein distance for the behaviour of Stochastic Gradient Langevin Dynamics (SGLD) algorithms. We allow the estimation of gradients to be performed even in the presence of \emph{dependent} data streams. Our convergence estimates are sharper and \emph{uniform} in the number of iterations, in contrast to those in…
Tencent AI Lab（腾讯人工智能实验室）；HongKong University of Science and Technology（香港科技大学）；Zhejiang University（浙江大学）
Recently, generalization bounds of the non-convex empirical risk minimization paradigm using Stochastic Gradient Langevin Dynamics (SGLD) have been extensively studied. Several theoretical frameworks have been presented to study this problem from different perspectives, such as information theory and stability. In this paper, we present a unified view from privacy leakage analysis to investigate the generalization bounds of SGLD, along with a theoretical framework for re-deriving previous results in a succinct manner. Aside from theoretical findings, we conduct various numerical studies to emp…
Error estimates between SGD with momentum and underdamped Langevin diffusion
Laboratoire de Mathematiques Blaise Pascal, CNRS-UMR 6620, Universite Clermont-Auvergne（克莱蒙奥弗涅大学）；University of Macau（澳门大学）；UM Zhuhai Research Institute（澳门大学珠海研究院）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Beijing International Center for Mathematical Research (BICMR), Peking University（北京大学北京国际数学研究中心）
UNIST InnoCORE AI-Space Solar Initiative, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（UNIST InnoCORE AI-Space Solar Initiative，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；Artificial Intelligence Graduate School, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（人工智能研究生院，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；Department of Industrial Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（工业工程系，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；School of Mathematics, University of Edinburgh, Edinburgh, United Kingdom（爱丁堡大学数学学院，爱丁堡，英国）；Department of Mathematics, National Technical University of Athens, Athens, Greece（雅典国家技术大学数学系，雅典，希腊）；Archimedes, Athena Research and Innovation Centre, Marousi, Greece（Archimedes，雅典研究与创新中心，Marousi，希腊）
Laboratoire de Probabilités, Statistique et Modélisation（概率、统计与建模实验室）；Sorbonne Université（索邦大学）
Training a very deep neural network is a challenging task, as the deeper a neural network is, the more non-linear it is. We compare the performances of various preconditioned Langevin algorithms with their non-Langevin counterparts for the training of neural networks of increasing depth. For shallow neural networks, Langevin algorithms do not lead to any improvement, however the deeper the network is and the greater are the gains provided by Langevin algorithms. Adding noise to the gradient descent allows to escape from local traps, which are more frequent for very deep neural networks. Follow…
Application of Langevin Dynamics to Advance the Quantum Natural Gradient Optimization Algorithm
National Science Center "Kharkiv Institute of Physics and Technology"（国家科学中心"基尔基亚物理与技术研究所"）；V.N. Karazin Kharkiv National University（V.N. 卡拉津基尔基亚国立大学）；Università della Svizzera italiana（瑞士意大利大学）
The Chinese University of Hong Kong（香港中文大学）；Rutgers University（罗格斯大学）；Florida State University（佛罗里达州立大学）
Stochastic gradient Hamiltonian Monte Carlo (SGHMC) is a variant of stochastic gradient with momentum where a controlled and properly scaled Gaussian noise is added to the stochastic gradients to steer the iterates towards a global minimum. Many works reported its empirical success in practice for solving stochastic non-convex optimization problems, in particular it has been observed to outperform overdamped Langevin Monte Carlo-based methods such as stochastic gradient Langevin dynamics (SGLD) in many applications. Although asymptotic global convergence properties of SGHMC are well known, its…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Department of Mathematics（数学系）；Madison, WI 53706（威斯康星州麦迪逊市 53706（注：该地址非机构，此处仅按要求提取原文中明确出现的机构相关内容））
Sampling from a log-concave distribution function is one core problem that has wide applications in Bayesian statistics and machine learning. While most gradient free methods have slow convergence rate, the Langevin Monte Carlo (LMC) that provides fast convergence requires the computation of gradients. In practice one uses finite-differencing approximations as surrogates, and the method is expensive in high-dimensions. A natural strategy to reduce computational cost in each iteration is to utilize random gradient approximations, such as random coordinate descent (RCD) or simultaneous perturbat…
Yandex；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；HSE University（俄罗斯国立高等经济学院）
This paper introduces Stochastic Gradient Langevin Boosting (SGLB) - a powerful and efficient machine learning framework that may deal with a wide range of loss functions and has provable generalization guarantees. The method is based on a special form of the Langevin diffusion equation specifically designed for gradient boosting. This allows us to theoretically guarantee the global convergence even for multimodal loss functions, while standard gradient boosting algorithms can guarantee only local optimum. We also empirically show that SGLB outperforms classic gradient boosting when applied to…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
We consider the task of sampling with respect to a log concave probability distribution. The potential of the target distribution is assumed to be composite, \textit{i.e.}, written as the sum of a smooth convex term, and a nonsmooth convex term possibly taking infinite values. The target distribution can be seen as a minimizer of the Kullback-Leibler divergence defined on the Wasserstein space (\textit{i.e.}, the space of probability measures). In the first part of this paper, we establish a strong duality result for this minimization problem. In the second part of this paper, we use the duali…
We propose an interacting contour stochastic gradient Langevin dynamics (ICSGLD) sampler, an embarrassingly parallel multiple-chain contour stochastic gradient Langevin dynamics (CSGLD) sampler with efficient interactions. We show that ICSGLD can be theoretically more efficient than a single-chain CSGLD with an equivalent computational budget. We also present a novel random-field function, which facilitates the estimation of self-adapting parameters in big data and obtains free mode explorations. Empirically, we compare the proposed algorithm with popular benchmark methods for posterior sampli…
Helsinki Institute for Information Technology（赫尔辛基信息技术研究所）；Aalto University（阿尔托大学）；F-Secure（芬安全公司）；University of Manchester（曼彻斯特大学）
Stochastic gradient MCMC methods, such as stochastic gradient Langevin dynamics (SGLD), employ fast but noisy gradient estimates to enable large-scale posterior sampling. Although we can easily extend SGLD to distributed settings, it suffers from two issues when applied to federated non-IID data. First, the variance of these estimates increases significantly. Second, delaying communication causes the Markov chains to diverge from the true posterior even for very simple models. To alleviate both these problems, we propose conducive gradients, a simple mechanism that combines local likelihood ap…
Rutgers Business School（罗格斯大学商学院）；The Chinese University of Hong Kong（香港中文大学）；Florida State University（佛罗里达州立大学）
Stochastic gradient Langevin dynamics (SGLD) and stochastic gradient Hamiltonian Monte Carlo (SGHMC) are two popular Markov Chain Monte Carlo (MCMC) algorithms for Bayesian inference that can scale to large datasets, allowing to sample from the posterior distribution of the parameters of a statistical model given the input data and the prior distribution over the model parameters. However, these algorithms do not apply to the decentralized learning setting, when a network of agents are working collaboratively to learn the parameters of a statistical model without sharing their individual data …
University of Cambridge（剑桥大学）；ETH Zürich（苏黎世联邦理工学院）
Stochastic gradient Markov Chain Monte Carlo algorithms are popular samplers for approximate inference, but they are generally biased. We show that many recent versions of these methods (e.g. Chen et al. (2014)) cannot be corrected using Metropolis-Hastings rejection sampling, because their acceptance probability is always zero. We can fix this by employing a sampler with realizable backwards trajectories, such as Gradient-Guided Monte Carlo (Horowitz, 1991), which generalizes stochastic gradient Langevin dynamics (Welling and Teh, 2011) and Hamiltonian Monte Carlo. We show that this sampler c…
We analyse the privacy leakage of noisy stochastic gradient descent by modeling Rényi divergence dynamics with Langevin diffusions. Inspired by recent work on non-stochastic algorithms, we derive similar desirable properties in the stochastic setting. In particular, we prove that the privacy loss converges exponentially fast for smooth and strongly convex objectives under constant step size, which is a significant improvement over previous DP-SGD analyses. We also extend our analysis to arbitrary sequences of varying step sizes and derive new utility bounds. Last, we propose an implementation …
Bayesian deep learning offers a principled way to address many issues concerning safety of artificial intelligence (AI), such as model uncertainty,model interpretability, and prediction bias. However, due to the lack of efficient Monte Carlo algorithms for sampling from the posterior of deep neural networks (DNNs), Bayesian deep learning has not yet powered our AI system. We propose a class of adaptive stochastic gradient Markov chain Monte Carlo (SGMCMC) algorithms, where the drift function is biased to enhance escape from saddle points and the bias is adaptively adjusted according to the gra…
