arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06500cs.LG
收藏

微 canonical 动力学能否利用小批量梯度噪声?

Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?

Department of Statistics, LMU Munich(统计系,慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Department of Physics, University of California, Berkeley(伯克利大学物理系);Department of Astronomy, Tsinghua University(清华大学天文系);Physics Division, Lawrence Berkeley National Lab(伯克利国家实验室物理部)

Emanuel Sommer, Kangning Diao, Jakob Robnik, Uros Seljak, David Rügamer

AI总结:

本文研究了微 canonical 动力学能否有效利用小批量梯度噪声,提出了一种梯度噪声预条件化方案和能量方差基于的自适应调节器,从而开发出一种鲁棒且可扩展的微 canonical 采样器,实现了在高维推断任务中的最佳性能。

摘要:

将推断方法如马尔可夫链蒙特卡罗扩展到高维模型仍然是贝叶斯深度学习中的核心挑战。一个有前景的最新提案,微 canonical 动力学蒙特卡罗,在广泛的问题上展示了最先进的性能。然而,其对完整数据集梯度的依赖使其在大规模问题中成本过高。本文解决了一个根本性问题:微 canonical 动力学能否有效利用小批量梯度噪声?我们提供了该问题的第一个系统研究,建立了随机梯度微 canonical 动力学的新型连续时间理论分析。我们揭示了两种关键的失败模式:由于各向异性梯度噪声导致的理论偏置和复杂高维后验中的数值不稳定性。为解决这些问题,我们提出了一种原理性的梯度噪声预条件化方案,已证明能显著减少这种偏置,并开发了一种新的基于能量方差的自适应调节器,自动化步长选择并动态告知数值保护措施。所得到的算法是一种鲁棒且可扩展的微 canonical 采样器,能够在具有挑战性的高维推断任务如贝叶斯神经网络中实现最先进的性能。结合最近的集合技术,我们的工作解锁了一种新的随机微 canonical 动力学集合(SMILE)采样器类,用于大规模贝叶斯推断。

英文摘要:

Scaling inference methods such as Markov chain Monte Carlo to high-dimensional models remains a central challenge in Bayesian deep learning. A promising recent proposal, microcanonical Langevin Monte Carlo, has shown state-of-the-art performance across a wide range of problems. However, its reliance on full-dataset gradients makes it prohibitively expensive for large-scale problems. This paper addresses a fundamental question: Can microcanonical dynamics effectively leverage mini-batch gradient noise? We provide the first systematic study of this problem, establishing a novel continuous-time theoretical analysis of stochastic-gradient microcanonical dynamics. We reveal two critical failure modes: a theoretically derived bias due to anisotropic gradient noise and numerical instabilities in complex high-dimensional posteriors. To tackle these issues, we propose a principled gradient noise preconditioning scheme shown to significantly reduce this bias and develop a novel, energy-variance-based adaptive tuner that automates step size selection and dynamically informs numerical guardrails. The resulting algorithm is a robust and scalable microcanonical Monte Carlo sampler that achieves state-of-the-art performance on challenging high-dimensional inference tasks like Bayesian neural networks. Combined with recent ensemble techniques, our work unlocks a new class of stochastic microcanonical Langevin ensemble (SMILE) samplers for large-scale Bayesian inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑