arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.18707cs.LGstat.ML
收藏

有限神经网络作为高斯过程的混合:从可证明的误差界到先验选择

Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection

Delft Center for Systems and Control (DCSC), Delft University of Technology, The Netherlands(代尔夫特系统与控制中心(DCSC),代尔夫特理工大学,荷兰);School of Computer Science and Statistics, Trinity College Dublin, Ireland(三一学院都柏林计算机科学与统计学系,爱尔兰);Departments of Aerospace Engineering Sciences and Computer Science, University of Colorado Boulder, USA(科罗拉多大学波德分校航空航天工程科学系与计算机科学系,美国);The Italian Institute of Artificial Intelligence (AI4I), Italy(意大利人工智能研究所(AI4I),意大利)

Steven Adams, Andrea Patanè, Morteza Lahijanian, Luca Laurenti

AI总结:

本文提出了一种算法框架,通过高斯过程混合近似有限宽度和深度的神经网络,并在误差界内进行逼近,同时探讨了如何利用该方法调整神经网络参数以模仿给定高斯过程的函数行为,如贝叶斯推断中的先验选择。

摘要:

无限宽或深的神经网络(NNs)具有独立同分布(i.i.d.)参数已被证明等价于高斯过程。由于高斯过程的有利性质,这种等价性常用于分析神经网络并导致了多年来的各种突破。然而,神经网络和高斯过程仅在极限情况下等价;在有限情况下目前尚无方法可用以近似训练好的神经网络为高斯模型并具有误差界的近似误差界。在本文中,我们提出了一种算法框架,用于近似有限宽度和深度的神经网络,并且参数不一定是i.i.d.,通过高斯过程的混合来近似,并在近似误差上具有误差界。特别是,我们考虑了Wasserstein距离来量化概率模型的接近程度,并依靠最优传输和高斯过程的工具,我们迭代地将神经网络的每一层输出分布近似为高斯过程的混合。关键的是,对于任何NN和ε>0,我们的方法能够返回一个混合的高斯过程,在有限的输入点集上与NN在ε范围内接近。此外,我们依靠所得到的误差界可微性,展示了我们的方法如何用于调整神经网络参数以模仿给定高斯过程的函数行为,例如在贝叶斯推断中的先验选择。我们通过回归和分类问题的不同神经网络架构进行了实证研究,实验结果突显了我们的结果如何代表理解神经网络预测和正式量化其不确定性的重要一步。

英文摘要:

Infinitely wide or deep neural networks (NNs) with independent and identically distributed (i.i.d.) parameters have been shown to be equivalent to Gaussian processes. Because of the favorable properties of Gaussian processes, this equivalence is commonly employed to analyze neural networks and has led to various breakthroughs over the years. However, neural networks and Gaussian processes are equivalent only in the limit; in the finite case there are currently no methods available to approximate a trained neural network with a Gaussian model with bounds on the approximation error. In this work, we present an algorithmic framework to approximate a neural network of finite width and depth, and with not necessarily i.i.d. parameters, with a mixture of Gaussian processes with error bounds on the approximation error. In particular, we consider the Wasserstein distance to quantify the closeness between probabilistic models and, by relying on tools from optimal transport and Gaussian processes, we iteratively approximate the output distribution of each layer of the neural network as a mixture of Gaussian processes. Crucially, for any NN and $ε>0$ our approach is able to return a mixture of Gaussian processes that is $ε$-close to the NN at a finite set of input points. Furthermore, we rely on the differentiability of the resulting error bound to show how our approach can be employed to tune the parameters of a NN to mimic the functional behavior of a given Gaussian process, e.g., for prior selection in the context of Bayesian inference. We empirically investigate the effectiveness of our results on both regression and classification problems with various neural network architectures. Our experiments highlight how our results can represent an important step towards understanding neural network predictions and formally quantifying their uncertainty.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑