The paper gives a bound on the generalization error of the Gibbs algorithm, which recovers known data-independent bounds for the high temperature range and extends to the low-temperature range, where generalization depends critically on the data-dependent loss-landscape. It is shown, that with high probability the generalization error of a single hypothesis drawn from the Gibbs posterior decreases with the total prior volume of all hypotheses with similar or smaller empirical error. This gives theoretical support to the belief in the benefit of flat minima. The zero temperature limit is discussed and the bound is extended to a class of similar stochastic algorithms.
Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measures
Univ Rennes(雷恩大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);IRISA(雷恩信息与自动化研究所);Université Jean Monnet Saint-Etienne(圣艾蒂安让·莫奈大学);Institut d Optique Graduate School(光学研究所研究生院);Laboratoire Hubert Curien(于贝尔·居里实验室);Institut Universitaire de France (IUF)(法国大学研究院);ServiceNow Research(ServiceNow研究院)
Accelerated Markov Chain Monte Carlo Using Adaptive Weighting Scheme
School of Computer Science and Technology, North University of China(中北大学计算机科学与技术学院);School of Semiconductor and Physics, North University of China(中北大学半导体与物理学院)
PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory
Department of Statistics, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校统计系);Department of Mathematics, University of Maryland, College Park(马里兰大学学院市数学系)
How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?
University of Florida(佛罗里达大学);University College London(伦敦大学学院);National University of Singapore(新加坡国立大学);Cornell University(康奈尔大学);The Alan Turing Institute(艾伦·图灵研究所)