While generalized linear mixed models are a fundamental tool in applied statistics, many specifications, such as those involving categorical factors with many levels or interaction terms, can be computationally challenging to estimate due to the need to compute or approximate high-dimensional integrals. Variational inference is a popular way to perform such computations, especially in the Bayesian context. However, naive use of such methods can provide unreliable uncertainty quantification. We show that this is indeed the case for mixed models, proving that standard mean-field variational inference dramatically underestimates posterior uncertainty in high-dimensions. We then show how appropriately relaxing the mean-field assumption leads to methods whose uncertainty quantification does not deteriorate in high-dimensions, and whose total computational cost scales linearly with the number of parameters and observations. Our theoretical and numerical results focus on mixed models with Gaussian or binomial likelihoods, and rely on connections to random graph theory to obtain sharp high-dimensional asymptotic analysis. We also provide generic results, which are of independent interest, relating the accuracy of variational inference to the convergence rate of the corresponding coordinate ascent algorithm that is used to find it. Our proposed methodology is implemented in the R package, see https://github.com/mgoplerud/vglmer . Numerical results with simulated and real data examples illustrate the favourable computation cost versus accuracy trade-off of our approach compared to various alternatives.
University of Toronto(多伦多大学);Vector Institute(向量研究所)
变分贝叶斯神经网络将深度学习的灵活性与贝叶斯不确定性估计相结合。不幸的是,在廉价但简单的变分族(例如~完全因式分解的)或昂贵且复杂的推理过程。我们表明,自然梯度上升与自适应权重噪声隐式地适合变分后验最大化的证据下限(ELBO)。这种见解使我们能够分别使用自然梯度、Adam和K-FAC的噪声版本来训练全协方差、全因子分解或矩阵变量高斯变分后验,从而可以扩展到现代规模的ConvNets。在标准回归基准上,我们的噪声K-FAC算法比现有方法更好地预测和匹配Hamilton Monte Carlo的预测方差。其改进的不确定性估计导致主动学习中更有效的探索,以及强化学习的内在动机。
Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs
Department of Statistics University of British Columbia(统计学系大学哥伦比亚大学);CERMICS Laboratory Ecole des Ponts ParisTech(巴黎高等理工学院CERMICS实验室);Center for Computational Mathematics Flatiron Institute(Flatiron研究所计算数学中心)
Recovering Latent Structures after Variational Bayesian Variable Selection: Fit Assessment and Factor-Number Selection in Partially Exploratory Factor Analysis
Faculty of Education, The University of Hong Kong(香港大学教育学院);Department of Curriculum and Instruction, The Education University of Hong Kong(香港教育大学课程与教学系)