We study the problem of non-convex optimization using Stochastic Gradient Langevin Dynamics (SGLD). SGLD is a natural and popular variation of stochastic gradient descent where at each step, appropriately scaled Gaussian noise is added. To our knowledge, the only strategy for showing global convergence of SGLD on the loss function is to show that SGLD can sample from a stationary distribution which assigns larger mass when the function is small (the Gibbs measure), and then to convert these guarantees to optimization results.
We employ a new strategy to analyze the convergence of SGLD to global minima, based on Lyapunov potentials and optimization. We convert the same mild conditions from previous works on SGLD into geometric properties based on Lyapunov potentials. This adapts well to the case with a stochastic gradient oracle, which is natural for machine learning applications where one wants to minimize population loss but only has access to stochastic gradients via minibatch training samples. Here we provide 1) improved rates in the setting of previous works studying SGLD for optimization, 2) the first finite gradient complexity guarantee for SGLD where the function is Lipschitz and the Gibbs measure defined by the function satisfies a Poincaré Inequality, and 3) prove if continuous-time Langevin Dynamics succeeds for optimization, then discrete-time SGLD succeeds under mild regularity assumptions.
Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)（人工智能基础研究实验室（BRAIn Lab））；Innopolis University（因诺普利斯大学）；P.N. Lebedev Physical Institute of the Russian Academy of Sciences（俄罗斯科学院皮亚琴佐·列别杰夫物理研究所）
Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院LTCI实验室）；Department of Statistics, University of Oxford（牛津大学统计系）；Department of Mathematics, Florida State University（佛罗里达州立大学数学系）；Department of Management Science and Information Systems, Rutgers Business School（罗格斯大学商学院管理科学与信息系统系）
带动量的随机梯度下降（SGDm）是深度学习中最流行的优化算法之一。虽然有丰富的SGDm理论用于凸问题，但该理论在深度学习的背景下发展得相当少，其中问题是非凸的，并且梯度噪声可能表现出重尾行为，正如最近的研究所观察到的那样。在这项研究中，我们考虑了一个\emph{continuous-time}变形的SGDm，称为欠阻尼朗之万动力学（ULD），并研究其渐近性质下的重尾扰动。支持最近的研究从统计物理，我们认为在理论上和经验上，这种扰动的重尾可以导致偏见，即使当步长很小，在这个意义上说，\emph{the optima of stationary distribution}的动态可能不匹配\emph{the optima of the cost function to be optimized}。作为补救措施，我们开发了一个新的框架，我们将其命名为\emph{fractional} ULD（FULD），并证明FULD针对所谓的吉布斯分布，其最优值与原始成本的最优值完全匹配。我们观察到，FULD的欧拉离散化与\emph{natural gradient}方法和\emph{gradient clipping}具有值得注意的算法相似性，为理解它们在深度学习中的作用带来了新的视角。我们支持我们的理论与合成模型和神经网络上进行的实验。
Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning
Department of Electrical and Computer Engineering（电气与计算机工程系）；Georgia Institute of Technology（佐治亚理工学院）；Department of Mathematics（数学系）；Massachusetts Institute of Technology（麻省理工学院）
Generative Modeling through Koopman Spectral Analysis: An Operator-Theoretic Perspective
Center for Science Adventure；Collaborative Research Advancement, Graduate School of Science, Kyoto University, Kyoto, Japan；Massachusetts Institute of Technology, Cambridge, MA；LinkedIn Corporation, Sunnyvale, CA；Dynamic Systems Lab, University College London, London, UK；Graduate School of Information Science；Technology, The University of Osaka, Osaka, Japan；RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
UNIST InnoCORE AI-Space Solar Initiative, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（UNIST InnoCORE AI-Space Solar Initiative，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；Artificial Intelligence Graduate School, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（人工智能研究生院，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；Department of Industrial Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（工业工程系，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；School of Mathematics, University of Edinburgh, Edinburgh, United Kingdom（爱丁堡大学数学学院，爱丁堡，英国）；Department of Mathematics, National Technical University of Athens, Athens, Greece（雅典国家技术大学数学系，雅典，希腊）；Archimedes, Athena Research and Innovation Centre, Marousi, Greece（Archimedes，雅典研究与创新中心，Marousi，希腊）
