arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.15191cs.LGstat.ML
收藏

SGD是贝叶斯采样器吗?差不多

Is SGD a Bayesian sampler? Well, almost

University of Oxford(牛津大学)

Chris Mingard, Guillermo Valle-Pérez, Joar Skalse, Ard A. Louis

摘要:

过度参数化的深度神经网络(DNN)具有高度的表达能力,因此原则上可以生成几乎任何适合零误差训练数据集的函数。这些函数中的绝大多数在看不见的数据上表现不佳,但在实践中,DNN通常具有非常好的泛化能力。这一成功表明,经过训练的DNN必须对泛化误差低的函数具有很强的归纳偏见。在这里,我们通过计算一系列架构和数据集的概率$P_{SGD}(f\mid S)$来实证研究这种归纳偏差,该概率是用随机梯度下降(SGD)或其变体之一训练的过参数化DNN收敛于与训练集$S$一致的函数$f$。我们还使用高斯过程来估计贝叶斯后验概率$P_B(f\mid S)$,DNN表示$f$随机采样的参数,条件是$S$。我们的主要发现是,$P_{SGD}(f\mid S)$相关性非常好,与$P_B(f\mid S)$和$P_B(f\mid S)$是强烈偏向于低错误和低复杂性的功能。这些结果意味着参数函数映射(决定$P_B(f\mid S)$)中的强归纳偏差,而不是SGD的特殊属性,是DNN在过参数化机制中推广得如此之好的主要解释。虽然我们的研究结果表明,贝叶斯后$P_B(f\mid S)$是$P_{SGD}(f\mid S)$的一阶行列式,仍然存在对超参数调整敏感的二阶差异。基于$P_{SGD}(f\mid S)$和/或$P_B(f\mid S)$的函数概率图可以揭示架构或超参数设置(如批量大小、学习率和优化器选择)的变化如何影响DNN性能。

英文摘要:

Overparameterised deep neural networks (DNNs) are highly expressive and so can, in principle, generate almost any function that fits a training dataset with zero error. The vast majority of these functions will perform poorly on unseen data, and yet in practice DNNs often generalise remarkably well. This success suggests that a trained DNN must have a strong inductive bias towards functions with low generalisation error. Here we empirically investigate this inductive bias by calculating, for a range of architectures and datasets, the probability $P_{SGD}(f\mid S)$ that an overparameterised DNN, trained with stochastic gradient descent (SGD) or one of its variants, converges on a function $f$ consistent with a training set $S$. We also use Gaussian processes to estimate the Bayesian posterior probability $P_B(f\mid S)$ that the DNN expresses $f$ upon random sampling of its parameters, conditioned on $S$. Our main findings are that $P_{SGD}(f\mid S)$ correlates remarkably well with $P_B(f\mid S)$ and that $P_B(f\mid S)$ is strongly biased towards low-error and low complexity functions. These results imply that strong inductive bias in the parameter-function map (which determines $P_B(f\mid S)$), rather than a special property of SGD, is the primary explanation for why DNNs generalise so well in the overparameterised regime. While our results suggest that the Bayesian posterior $P_B(f\mid S)$ is the first order determinant of $P_{SGD}(f\mid S)$, there remain second order differences that are sensitive to hyperparameter tuning. A function probability picture, based on $P_{SGD}(f\mid S)$ and/or $P_B(f\mid S)$, can shed new light on the way that variations in architecture or hyperparameter settings such as batch size, learning rate, and optimiser choice, affect DNN performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑