Algorithm-dependent generalization error bounds are central to statistical learning theory. A learning algorithm may use a large hypothesis space, but the limited number of iterations controls its model capacity and generalization error. The impacts of stochastic gradient methods on generalization error for non-convex learning problems not only have important theoretical consequences, but are also critical to generalization errors of deep learning.
In this paper, we study the generalization errors of Stochastic Gradient Langevin Dynamics (SGLD) with non-convex objectives. Two theories are proposed with non-asymptotic discrete-time analysis, using Stability and PAC-Bayesian results respectively. The stability-based theory obtains a bound of $O\left(\frac{1}{n}L\sqrt{βT_k}\right)$, where $L$ is uniform Lipschitz parameter, $β$ is inverse temperature, and $T_k$ is aggregated step sizes. For PAC-Bayesian theory, though the bound has a slower $O(1/\sqrt{n})$ rate, the contribution of each step is shown with an exponentially decaying factor by imposing $\ell^2$ regularization, and the uniform Lipschitz constant is also replaced by actual norms of gradients along trajectory. Our bounds have no implicit dependence on dimensions, norms or other capacity measures of parameter, which elegantly characterizes the phenomenon of "Fast Training Guarantees Generalization" in non-convex settings. This is the first algorithm-dependent result with reasonable dependence on aggregated step sizes for non-convex learning, and has important implications to statistical learning aspects of stochastic gradient methods in complicated models such as deep learning.
Dimension-free convergence rates for gradient Langevin dynamics in RKHS
CREST, ENSAE, IP Paris（巴黎理工学院国立统计与经济管理学院经济与统计研究中心）；The University of Tokyo（东京大学）；INRIA, Université Paris-Saclay（巴黎萨克雷大学法国国家信息与自动化研究所）；Riken AIP（理化学研究所先进智能项目中心）
Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration
The Chinese University of Hong Kong（香港中文大学）；Rutgers University（罗格斯大学）；Florida State University（佛罗里达州立大学）
随机梯度Hamilton Monte Carlo（SGHMC）是一种动量随机梯度算法，在随机梯度中加入受控的、适当缩放的高斯噪声，使迭代过程趋于全局极小值。许多工作报告了其在解决随机非凸优化问题的实践中的经验成功，特别是它已被观察到优于过阻尼Langevin蒙特卡罗方法，如随机梯度Langevin动力学（SGLD）在许多应用中。虽然SGHMC的渐近整体收敛性质是众所周知的，其有限时间性能还没有得到很好的理解。在这项工作中，我们研究了两个变种的SGHMC的基础上两个替代离散的欠阻尼朗之万扩散。我们提供了有限时间性能界的整体收敛的SGHMC解决显式常数随机非凸优化问题的变种。我们的研究结果导致人口和经验风险最小化问题的非渐近保证。对于一个固定的目标精度水平，对一类非凸问题，我们得到的复杂性界SGHMC可以比SGLD更紧。这些结果表明，在全局非凸优化的背景下，动量加速是可能的。
Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms
Alfréd Rényi Institute of Mathematics（阿尔弗雷德·雷尼数学研究所）；The University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）；National Technical University of Athens（雅典国立技术大学）
泛化误差（也称为样本外误差）衡量从训练数据学习的假设推广到以前未见过的数据的程度。证明紧的泛化误差界是统计学习理论的核心问题。在本文中，我们获得了泛化误差界学习一般的非凸目标，这是近年来引起了极大的关注。我们开发了一个新的框架，称为贝叶斯稳定性，证明算法依赖的泛化误差界。新的框架结合了PAC贝叶斯理论和算法稳定性的概念。应用贝叶斯稳定性方法，我们获得了随机梯度Langevin动力学（SGLD）和其他几种噪声梯度方法（例如，具有动量、小批量和加速、熵-SGD）。我们的结果恢复了Mou et al.（2018）的最近结果（通常比Mou et al.（2018）更严格），并改进了Pensia et al.（2018）的结果。我们的实验表明，我们的数据依赖边界可以区分随机标记的数据和正常数据，这为Zhang et al.（2017 a）中观察到的有趣现象提供了解释。我们还研究了总损失是有界损失和一个额外的\ell_2正则化项的总和的设置。通过对任意时刻的参数分布建立一个新的Log-Sobolev不等式，我们得到了连续Langevin动力学的新的推广界。当过程的噪声水平不小时，我们的新边界更令人满意，并且即使当T趋于无穷大时也不会变得空洞。
Stability of Primal-Dual Gradient Flow Dynamics for Multi-Block Convex Optimization Problems
Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California（明希德电气与计算机工程学院，南加州大学）；Department of Electrical Engineering (ESAT-STADIUS), KU Leuven（电气工程系（ESAT-STADIUS），鲁汶大学）
Langevin dynamics based algorithm e-TH$\varepsilon$O POULA for stochastic optimization problems with discontinuous stochastic gradient
UNIST（蔚山科学技术院）；Nanyang Technological University（南洋理工大学）；The University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（阿兰·图灵研究所）；National Technical University of Athens（雅典国家技术大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
