朗之万扩散形式的随机微分方程因其在贝叶斯采样算法和机器学习优化中的基础作用而受到广泛关注。在后者中,它们作为训练过参数化模型时随机梯度流的概念模型。然而,现有文献通常假设势函数是光滑的,其梯度即为漂移项。但在许多问题中,势函数并非连续可微,因此漂移项并非处处满足利普希茨连续性,回归问题中的鲁棒损失和修正线性单元(Rectified Linear Units)就是典型例子。本文在适用于机器学习场景的假设下,给出了朗之万型随机微分包含的流性质与渐近性质的若干基础结果。具体而言,我们证明了解的强存在性,以及正则自由能泛函的渐近最小化特性。
英文摘要:
Stochastic differential equations of Langevin-diffusion form have received significant attention, thanks to their foundational role in both Bayesian sampling algorithms and optimization in machine learning. In the latter, they serve as a conceptual model of the stochastic gradient flow in training over-parameterized models. However, the literature typically assumes smoothness of the potential, whose gradient is the drift term. Nevertheless, there are many problems for which the potential function is not continuously differentiable, and hence the drift is not Lipschitz continuous everywhere. This is exemplified by robust losses and Rectified Linear Units in regression problems. In this paper, we show some foundational results regarding the flow and asymptotic properties of Langevin-type Stochastic Differential Inclusions under assumptions appropriate to the machine-learning settings. In particular, we show strong existence of the solution, as well as an asymptotic minimization of the canonical free-energy functional.
Langevin dynamics based algorithm e-TH$\varepsilon$O POULA for stochastic optimization problems with discontinuous stochastic gradient
UNIST(蔚山科学技术院);Nanyang Technological University(南洋理工大学);The University of Edinburgh(爱丁堡大学);The Alan Turing Institute(阿兰·图灵研究所);National Technical University of Athens(雅典国家技术大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms
Alfréd Rényi Institute of Mathematics(阿尔弗雷德·雷尼数学研究所);The University of Edinburgh(爱丁堡大学);The Alan Turing Institute(艾伦·图灵研究所);National Technical University of Athens(雅典国立技术大学)