arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09048stat.MLcs.LGmath.PRmath.STstat.TH
收藏

使用变分推断训练的贝叶斯神经网络的中心极限定理

Central Limit Theorem for Bayesian Neural Network trained with Variational Inference

Laboratoire de Mathématiques Blaise Pascal(布莱斯·帕斯卡数学实验室);Université Clermont-Auvergne(克莱蒙奥弗涅大学);INRIA Lille(里尔国家计算机与自动化研究所);Centre de Mathématiques Appliquées(应用数学中心);École polytechnique(巴黎综合理工学院);CNRS(法国国家科学研究中心);Institut Universitaire de France(法国大学研究院)

Arnaud Descours, Tom Huix, Arnaud Guillin, Manon Michel, Éric Moulines, Boris Nectoux

AI总结:

本文研究变分推断训练的无限宽贝叶斯两层神经网络,推导三种训练方案的中心极限定理,揭示 Minimal VI 与其他方案的波动差异,并验证其计算效率优势。

摘要:

本文严格推导了无限宽度极限下、在回归任务上通过变分推断训练的贝叶斯两层神经网络的中心极限定理(CLT)。不同网络通过正则化证据下界的不同最大化方案进行训练:(i)理想化情形,即利用重参数化技巧精确估计一个多重高斯积分;(ii)使用 Monte Carlo 采样的小批量方案,通常称为 Bayes-by-Backprop;以及(iii)一种计算成本更低的算法,名为 Minimal VI。后一算法是最近借助在平均场极限层面获得的信息而提出的。针对这三种具有相同渐近极限的方案,大数定律已经得到严格证明。通过推导 CLT,本工作表明理想化方案与 Bayes-by-Backprop 方案具有相似的波动行为,而这不同于 Minimal VI 的波动行为。随后,数值实验说明:尽管 Minimal VI 方案具有更大的方差,但由于其在计算复杂度上的显著收益,它仍然更高效。

英文摘要:

In this paper, we rigorously derive Central Limit Theorems (CLT) for Bayesian two-layerneural networks in the infinite-width limit and trained by variational inference on a regression task. The different networks are trained via different maximization schemes of the regularized evidence lower bound: (i) the idealized case with exact estimation of a multiple Gaussian integral from the reparametrization trick, (ii) a minibatch scheme using Monte Carlo sampling, commonly known as Bayes-by-Backprop, and (iii) a computationally cheaper algorithm named Minimal VI. The latter was recently introduced by leveraging the information obtained at the level of the mean-field limit. Laws of large numbers are already rigorously proven for the three schemes that admits the same asymptotic limit. By deriving CLT, this work shows that the idealized and Bayes-by-Backprop schemes have similar fluctuation behavior, that is different from the Minimal VI one. Numerical experiments then illustrate that the Minimal VI scheme is still more efficient, in spite of bigger variances, thanks to its important gain in computational complexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑