Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?
Department of Statistics, LMU Munich(统计系,慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Department of Physics, University of California, Berkeley(伯克利大学物理系);Department of Astronomy, Tsinghua University(清华大学天文系);Physics Division, Lawrence Berkeley National Lab(伯克利国家实验室物理部)
Emanuel Sommer, Kangning Diao, Jakob Robnik, Uros Seljak, David Rügamer
Scaling inference methods such as Markov chain Monte Carlo to high-dimensional models remains a central challenge in Bayesian deep learning. A promising recent proposal, microcanonical Langevin Monte Carlo, has shown state-of-the-art performance across a wide range of problems. However, its reliance on full-dataset gradients makes it prohibitively expensive for large-scale problems. This paper addresses a fundamental question: Can microcanonical dynamics effectively leverage mini-batch gradient noise? We provide the first systematic study of this problem, establishing a novel continuous-time theoretical analysis of stochastic-gradient microcanonical dynamics. We reveal two critical failure modes: a theoretically derived bias due to anisotropic gradient noise and numerical instabilities in complex high-dimensional posteriors. To tackle these issues, we propose a principled gradient noise preconditioning scheme shown to significantly reduce this bias and develop a novel, energy-variance-based adaptive tuner that automates step size selection and dynamically informs numerical guardrails. The resulting algorithm is a robust and scalable microcanonical Monte Carlo sampler that achieves state-of-the-art performance on challenging high-dimensional inference tasks like Bayesian neural networks. Combined with recent ensemble techniques, our work unlocks a new class of stochastic microcanonical Langevin ensemble (SMILE) samplers for large-scale Bayesian inference.
Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks
LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of California Berkeley(加州大学伯克利分校);Eraneos Analytics Germany GmbH(埃拉诺斯分析德国有限公司);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)
Discrete Langevin Sampler via Wasserstein Gradient Flow
Georgia Tech(佐治亚理工学院);Google Research(谷歌研究院);University of Alberta(阿尔伯塔大学)
已知连续空间的基于梯度的MCMC采样器,例如朗之万蒙特卡罗(LMC),可以导出为梯度流的粒子版本,其最小化Wasserstein流形上的KL发散。这种采样器的优越效率激发了最近几次尝试将LMC推广到离散空间。然而,朗之万动力学的一个完整的原则性扩展到离散空间尚未实现,由于缺乏定义良好的梯度在样本空间。在这项工作中,我们展示了如何Wasserstein梯度流可以自然地推广到离散空间。鉴于所提出的配方,我们演示了如何离散模拟朗之万动力学可以随后开发。有了这个新的理解,我们揭示了最近的基于梯度的采样器在离散空间中可以通过选择特定的离散化作为特殊情况。更重要的是,该框架还允许推导新的算法,其中之一,\textit{Discrete Langevin Monte Carlo}(DLMC),是通过一个因式分解的估计的过渡矩阵。DLMC方法允许一个方便的并行实现和时间均匀采样,实现更大的跳跃距离。我们证明了DLMC在各种二进制和分类分布上的优势。
Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches
Department of Physics, Università degli Studi di Milano(米兰大学物理系);Department of Computing Sciences(计算科学系);Bocconi Institute for Data Science(博科尼数据科学研究所);Analytics (BIDSA), Bocconi University(分析(BIDSA),博科尼大学)
Breaking Reversibility Accelerates Langevin Dynamics for Global Non-Convex Optimization
Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Shatin, N.T. Hong Kong(系统工程与工程管理系,香港中文大学(深圳));Department of Management Science and Information Systems and the DIMACS Institute, Rutgers University, Piscataway, NJ-08854, United States of America(管理科学与信息系统系及DIMACS研究所,罗杰斯大学);Department of Mathematics, Florida State University, 1017 Academic Way, Tallahassee, FL-32306, United States of America(数学系,佛罗里达州立大学)
Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction
University of Tokyo(东京大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(矢量人工智能研究所);Kyushu Institute of Technology(九州工业大学)
Convex Analysis of the Mean Field Langevin Dynamics
Kyushu Institute of Technology(九州工业大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);The University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(向量人工智能研究所);The University of Tokyo(东京大学)
Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration
The Chinese University of Hong Kong(香港中文大学);Rutgers University(罗格斯大学);Florida State University(佛罗里达州立大学)
随机梯度Hamilton Monte Carlo(SGHMC)是一种动量随机梯度算法,在随机梯度中加入受控的、适当缩放的高斯噪声,使迭代过程趋于全局极小值。许多工作报告了其在解决随机非凸优化问题的实践中的经验成功,特别是它已被观察到优于过阻尼Langevin蒙特卡罗方法,如随机梯度Langevin动力学(SGLD)在许多应用中。虽然SGHMC的渐近整体收敛性质是众所周知的,其有限时间性能还没有得到很好的理解。在这项工作中,我们研究了两个变种的SGHMC的基础上两个替代离散的欠阻尼朗之万扩散。我们提供了有限时间性能界的整体收敛的SGHMC解决显式常数随机非凸优化问题的变种。我们的研究结果导致人口和经验风险最小化问题的非渐近保证。对于一个固定的目标精度水平,对一类非凸问题,我们得到的复杂性界SGHMC可以比SGLD更紧。这些结果表明,在全局非凸优化的背景下,动量加速是可能的。
Stochastic Gradient Hamiltonian Monte Carlo with Variance Reduction for Bayesian Inference
Tsinghua University(清华大学)
基于随机性的Monte Carlo抽样算法,如Langevin动力学和Hamilton Monte Carlo,是贝叶斯推理的重要方法。在大规模环境中,全梯度是负担不起的,因此在小批量上评估的随机梯度被用作替代。为了降低噪声随机梯度的高方差,Dubey等人[2016]将标准方差降低技术应用于随机梯度Langevin动力学,并获得了理论和实验上的改进。在本文中,我们将方差缩减技巧应用于哈密顿蒙特卡罗,并取得了更好的理论收敛结果相比,方差缩减的朗之万动力学。此外,我们将对称分裂格式应用到方差缩减的哈密顿蒙特卡罗算法中,进一步改进了理论结果。实验结果也与理论结果一致。正如我们的实验所示,在真实世界数据集上的贝叶斯回归和分类任务中,方差缩减的哈密顿蒙特卡罗比方差缩减的朗之万动力学表现出更好的性能。
Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms
Alfréd Rényi Institute of Mathematics(阿尔弗雷德·雷尼数学研究所);The University of Edinburgh(爱丁堡大学);The Alan Turing Institute(艾伦·图灵研究所);National Technical University of Athens(雅典国立技术大学)
Application of Langevin Dynamics to Advance the Quantum Natural Gradient Optimization Algorithm
National Science Center "Kharkiv Institute of Physics and Technology"(国家科学中心"基尔基亚物理与技术研究所");V.N. Karazin Kharkiv National University(V.N. 卡拉津基尔基亚国立大学);Università della Svizzera italiana(瑞士意大利大学)
Xingjian College, Tsinghua University(清华大学行健书院);University of California, Berkeley(加州大学伯克利分校);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院);Department of Engineering Mechanics, Tsinghua University(清华大学工程力学系);Center on Frontiers of Computing Studies, School of Computer Science, Peking University(北京大学计算机学院计算前沿研究中心)
Langevin dynamics based algorithm e-TH$\varepsilon$O POULA for stochastic optimization problems with discontinuous stochastic gradient
UNIST(蔚山科学技术院);Nanyang Technological University(南洋理工大学);The University of Edinburgh(爱丁堡大学);The Alan Turing Institute(阿兰·图灵研究所);National Technical University of Athens(雅典国家技术大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))