arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.12418cs.LGstat.ML
收藏

迈向深度贝叶斯模型的改进变分推断

Towards Improved Variational Inference for Deep Bayesian Models

University of Cambridge(剑桥大学)

Sebastian W. Ober

AI总结:

本文针对深度贝叶斯模型后验难处理与边际似然界不紧的问题,研究改进变分推断,提出参数/超参数划分、统一变分后验及利用对称性消除在 Gram 矩阵上推断的方法。

摘要:

深度学习在过去十年中带来了革命性变化,处于计算机视觉、自然语言处理和强化学习等众多任务非凡进展的前沿。然而,众所周知,通过最大似然估计训练的深度模型往往过于自信,并给出校准不良的预测。贝叶斯深度学习试图通过对模型参数设置先验来解决这一问题,然后将这些先验与似然相结合以执行后验推断。遗憾的是,对于深度模型而言,真实后验是难以处理的,迫使用户求助于近似方法。在本论文中,我们探索使用变分推断(VI)作为近似方法,因为它的独特之处在于能够同时近似后验并提供边际似然的下界。如果该下界足够紧,就可以用于优化超参数并促进模型选择。然而,这种能力在贝叶斯神经网络中很少得到充分利用,可能是因为实践中通常使用的近似后验缺乏有效约束边际似然所需的灵活性。因此,我们探索深度模型贝叶斯学习的三个方面:1)我们追问是否有必要对尽可能多的参数执行推断,或者将其中许多参数视为可优化的超参数是否合理;2)我们提出一种变分后验,为贝叶斯神经网络和深度高斯过程中的推断提供统一视角;3)我们展示如何通过以解析方式消除后验中的对称性,并在 Gram 矩阵而非特征上执行推断,来改进某些深度高斯过程模型中的 VI。我们希望我们的贡献将成为未来充分实现 VI 前景的垫脚石。

英文摘要:

Deep learning has revolutionized the last decade, being at the forefront of extraordinary advances in a wide range of tasks including computer vision, natural language processing, and reinforcement learning, to name but a few. However, it is well-known that deep models trained via maximum likelihood estimation tend to be overconfident and give poorly-calibrated predictions. Bayesian deep learning attempts to address this by placing priors on the model parameters, which are then combined with a likelihood to perform posterior inference. Unfortunately, for deep models, the true posterior is intractable, forcing the user to resort to approximations. In this thesis, we explore the use of variational inference (VI) as an approximation, as it is unique in simultaneously approximating the posterior and providing a lower bound to the marginal likelihood. If tight enough, this lower bound can be used to optimize hyperparameters and to facilitate model selection. However, this capacity has rarely been used to its full extent for Bayesian neural networks, likely because the approximate posteriors typically used in practice can lack the flexibility to effectively bound the marginal likelihood. We therefore explore three aspects of Bayesian learning for deep models: 1) we ask whether it is necessary to perform inference over as many parameters as possible, or whether it is reasonable to treat many of them as optimizable hyperparameters; 2) we propose a variational posterior that provides a unified view of inference in Bayesian neural networks and deep Gaussian processes; 3) we demonstrate how VI can be improved in certain deep Gaussian process models by analytically removing symmetries from the posterior, and performing inference on Gram matrices instead of features. We hope that our contributions will provide a stepping stone to fully realize the promises of VI in the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑