arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22015cs.LG
收藏

函数空间经验贝叶斯正则化与学生t先验

Function-Space Empirical Bayes Regularisation with Student's t Priors

Institute of Data and Information(数据与信息研究所);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院)

Pengcheng Hao, Ercan Engin Kuruoglu

AI总结:

本文提出ST-FS-EB方法,利用学生t先验改进函数空间正则化,提升贝叶斯深度学习在不确定性估计和分布偏移处理中的性能。

摘要:

贝叶斯深度学习(BDL)作为一种原理性方法,通过将深度神经网络与贝叶斯推断相结合,产生了可靠的不确定性估计。然而,选择具有信息量的先验分布仍然是重大挑战。已经提出了各种函数空间变分推断(FSVI)正则化方法,将有意义的先验分配给模型预测。然而,这些方法通常依赖高斯先验,无法捕捉神经网络输出中固有的重尾统计特性。相比之下,本文提出了一种新的函数空间经验贝叶斯正则化框架——称为ST-FS-EB,该框架在参数空间和函数空间中均使用重尾的学生t先验。此外,我们通过变分推断(VI)近似后验分布,诱导出基于蒙特卡洛(MC)dropout的证据下界(ELBO)目标。进一步,所提出的方法在各种基于VI的BDL基线中进行了评估,结果表明其在分布内预测、分布外(OOD)检测和处理分布偏移方面具有稳健的性能。

英文摘要:

Bayesian deep learning (BDL) has emerged as a principled approach to produce reliable uncertainty estimates by integrating deep neural networks with Bayesian inference, and the selection of informative prior distributions remains a significant challenge. Various function-space variational inference (FSVI) regularisation methods have been presented, assigning meaningful priors over model predictions. However, these methods typically rely on a Gaussian prior, which fails to capture the heavy-tailed statistical characteristics inherent in neural network outputs. By contrast, this work proposes a novel function-space empirical Bayes regularisation framework -- termed ST-FS-EB -- which employs heavy-tailed Student's $t$ priors in both parameter and function spaces. Also, we approximate the posterior distribution through variational inference (VI), inducing an evidence lower bound (ELBO) objective based on Monte Carlo (MC) dropout. Furthermore, the proposed method is evaluated against various VI-based BDL baselines, and the results demonstrate its robust performance in in-distribution prediction, out-of-distribution (OOD) detection and handling distribution shifts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑