Comparing competing mathematical models of complex natural processes is a shared goal among many branches of science. The Bayesian probabilistic framework offers a principled way to perform model comparison and extract useful metrics for guiding decisions. However, many interesting models are intractable with standard Bayesian methods, as they lack a closed-form likelihood function or the likelihood is computationally too expensive to evaluate. With this work, we propose a novel method for performing Bayesian model comparison using specialized deep learning architectures. Our method is purely simulation-based and circumvents the step of explicitly fitting all alternative models under consideration to each observed dataset. Moreover, it requires no hand-crafted summary statistics of the data and is designed to amortize the cost of simulation over multiple models and observable datasets. This makes the method particularly effective in scenarios where model fit needs to be assessed for a large number of datasets, so that per-dataset inference is practically infeasible.Finally, we propose a novel way to measure epistemic uncertainty in model comparison problems. We demonstrate the utility of our method on toy examples and simulated data from non-trivial models from cognitive science and single-cell neuroscience. We show that our method achieves excellent results in terms of accuracy, calibration, and efficiency across the examples considered in this work. We argue that our framework can enhance and enrich model-based analysis and inference in many fields dealing with computational models of natural processes. We further argue that the proposed measure of epistemic uncertainty provides a unique proxy to quantify absolute evidence even in a framework which assumes that the true data-generating model is within a finite set of candidate models.
Heidelberg University（海德堡大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
We propose a novel method for closed-form predictive distribution modeling with neural nets. In quantifying prediction uncertainty, we build on Evidential Deep Learning, which has been impactful as being both simple to implement and giving closed-form access to predictive uncertainty. We employ it to model aleatoric uncertainty and extend it to account also for epistemic uncertainty by converting it to a Bayesian Neural Net. While extending its uncertainty quantification capabilities, we maintain its analytically accessible predictive distribution model by performing progressive moment matchin…
University of Helsinki（赫尔辛基大学）；ELLIS Institute Finland, Aalto University（芬兰埃利斯研究所、阿尔托大学）；TU Dortmund University（多特蒙德技术大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；Independent Scientist（独立研究者）
As models of cognition grow in complexity and number of parameters, Bayesian inference with standard methods can become intractable, especially when the data-generating model is of unknown analytic form. Recent advances in simulation-based inference using specialized neural network architectures circumvent many previous problems of approximate Bayesian computation. Moreover, due to the properties of these special neural network estimators, the effort of training the networks via simulations amortizes over subsequent evaluations which can re-use the same network for multiple datasets and across…
We develop methods for efficient amortized approximate Bayesian inference over posterior distributions of probabilistic clustering models, such as Dirichlet process mixture models. The approach is based on mapping distributed, symmetry-invariant representations of cluster arrangements into conditional probabilities. The method parallelizes easily, yields iid samples from the approximate posterior of cluster assignments with the same computational cost of a single Gibbs sampler sweep, and can easily be applied to both conjugate and non-conjugate models, as training only requires samples from th…
Neural Architectures for Amortized Bayesian Inference: Statistical Foundations and Empirical Assessments
Department of Mathematics and Statistics, Indian Institute of Technology Kanpur（数学与统计学系，印度理工学院坎浦尔分校）；Marshall School of Business, University of Southern California（马歇尔商学院，南加州大学）
Stellenbosch University（斯泰伦博斯大学）；National Institute for Theoretical and Computational Sciences（国家理论与计算科学研究所）
Initial work on variational autoencoders assumed independent latent variables with simple distributions. Subsequent work has explored incorporating more complex distributions and dependency structures: including normalizing flows in the encoder network allows latent variables to entangle non-linearly, creating a richer class of distributions for the approximate posterior, and stacking layers of latent variables allows more complex priors to be specified for the generative model. This work explores incorporating arbitrary dependency structures, as specified by Bayesian networks, into VAEs. This…
We present a framework that enables estimation of low-dimensional sub-resolution reservoir properties directly from seismic data, without requiring the solution of a high dimensional seismic inverse problem. Our workflow is based on the Bayesian evidential learning approach and exploits learning the direct relation between seismic data and reservoir properties to efficiently estimate reservoir properties. The theoretical framework we develop allows incorporation of non-linear statistical models for seismic estimation problems. Uncertainty quantification is performed with Approximate Bayesian C…
Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation
Tübingen AI Center（图宾根人工智能中心）；Machine Learning in Science, Excellence Cluster Machine Learning, University of Tübingen（图宾根大学机器学习卓越集群科学机器学习系）；Department Empirical Inference, Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所经验推断系）
Uncertainty-Aware Surrogate-based Amortized Bayesian Inference for Computationally Expensive Models
Department of Stochastic Simulation and Safety Research for Hydrosystems（随机模拟与安全研究部门）；Cluster of Excellence SimTech（卓越中心SimTech）；University of Stuttgart（斯图加特大学）；Department of Statistics（统计学系）；TU Dortmund University（多特蒙德技术大学）
From Mice to Trains: Amortized Bayesian Inference on Graph Data
Department of Statistics, TU Dortmund University（统计系，多特蒙德技术大学）；School of Public Health, Imperial College London（公共卫生学院，伦敦帝国理工学院）；Department of Biology, University of Oxford（生物学系，牛津大学）；Department of Mechanical Engineering, Karlsruhe Institute of Technology（机械工程系，卡尔斯鲁厄理工学院）
Cluster of Excellence SimTech（SimTech卓越集群）；University of Stuttgart（斯图加特大学）；Heidelberg University（海德堡大学）；Cluster of Excellence STRUCTURES（STRUCTURES卓越集群）
Neural density estimators have proven remarkably powerful in performing efficient simulation-based Bayesian inference in various research domains. In particular, the BayesFlow framework uses a two-step approach to enable amortized parameter estimation in settings where the likelihood function is implicitly defined by a simulation program. But how faithful is such inference when simulations are poor representations of reality? In this paper, we conceptualize the types of model misspecification arising in simulation-based inference and systematically investigate the performance of the BayesFlow …
Radboud University（拉德堡德大学）；University of Utrecht（乌得勒支大学）；University of Amsterdam（阿姆斯特丹大学）
In this paper, we introduce a new form of amortized variational inference by using the forward KL divergence in a joint-contrastive variational loss. The resulting forward amortized variational inference is a likelihood-free method as its gradient can be sampled without bias and without requiring any evaluation of either the model joint distribution or its derivatives. We prove that our new variational loss is optimized by the exact posterior marginals in the fully factorized mean-field approximation, a property that is not shared with the more conventional reverse KL inference. Furthermore, w…
With the surge in the number of hyperparameters and training times of modern machine learning models, hyperparameter tuning is becoming increasingly expensive. However, after assessing 40 tuning methods systematically, we find that each faces certain limitations. In particular, methods that speed up tuning via knowledge transfer typically require the final performance of hyperparameters and do not focus on low-fidelity information. As we demonstrate empirically, this common practice is suboptimal and can incur an unnecessary use of resources. It is more cost-efficient to instead leverage low-f…
Department of Statistics（统计学系）；TU Dortmund University（多特蒙德技术大学）；Cluster of Excellence SimTech（卓越中心SimTech）；University of Stuttgart（斯图加特大学）；Institute for Visualisation and Interactive Systems（可视化与交互系统研究所）；Rensselaer Polytechnic Institute（拉特克利夫理工学院）
German Aerospace Center (DLR)（德国航空航天中心）；Institute for Simulation Intelligence（模拟智能研究所）
There is significant need for principled uncertainty reasoning in machine learning systems as they are increasingly deployed in safety-critical domains. A new approach with uncertainty-aware neural networks (NNs), based on learning evidential distributions for aleatoric and epistemic uncertainties, shows promise over traditional deterministic methods and typical Bayesian NNs, yet several important gaps in the theory and implementation of these networks remain. We discuss three issues with a proposed solution to extract aleatoric and epistemic uncertainties from regression-based neural networks…
This paper presents a fast algorithm for estimating hidden states of Bayesian state space models. The algorithm is a variation of amortized simulation-based inference algorithms, where a large number of artificial datasets are generated at the first stage, and then a flexible model is trained to predict the variables of interest. In contrast to those proposed earlier, the procedure described in this paper makes it possible to train estimators for hidden states by concentrating only on certain characteristics of the marginal posterior distributions and introducing inductive bias. Illustrations …
Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation
Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad（计算机科学与工程系，印度理工学院海得拉巴分校）；Jagiellonian University, Faculty of Mathematics and Computer Science（雅盖隆大学，数学与计算机科学学院）
Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust, and efficient measures of uncertainty are crucial. In this paper, we propose a novel method for training non-Bayesian NNs to estimate a continuous target as well as its associated evidence in order to learn both aleatoric and epistemic uncertainty. We accomplish this by placing evidential priors over the original Gaussian likelihood function and training the NN to infer the hyperparameters of the evidential distribution. We additionally impose priors during training such th…
Deargen Inc.（Deargen 公司）；Deargen USA Inc.（Deargen 美国公司）
The Evidential regression network (ENet) estimates a continuous target and its predictive uncertainty without costly Bayesian model averaging. However, it is possible that the target is inaccurately predicted due to the gradient shrinkage problem of the original loss function of the ENet, the negative log marginal likelihood (NLL) loss. In this paper, the objective is to improve the prediction accuracy of the ENet while maintaining its efficient uncertainty estimation by resolving the gradient shrinkage problem. A multi-task learning (MTL) framework, referred to as MT-ENet, is proposed to acco…
ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition
ELLIS Institute Finland（芬兰ELLIS研究所）；Department of Computer Science, Aalto University, Finland（芬兰阿尔托大学计算机科学系）；Department of Computer Science, University of Helsinki, Finland（芬兰赫尔辛基大学计算机科学系）；Department of Computer Science, University of Manchester, UK（英国曼彻斯特大学计算机科学系）
Ozyegin University（奥兹耶金大学）；US Army Research Lab（美国陆军研究实验室）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Deterministic neural nets have been shown to learn effective predictors on a wide range of machine learning problems. However, as the standard approach is to train the network to minimize a prediction loss, the resultant model remains ignorant to its prediction confidence. Orthogonally to Bayesian neural nets that indirectly infer prediction uncertainty through weight uncertainties, we propose explicit modeling of the same using the theory of subjective logic. By placing a Dirichlet distribution on the class probabilities, we treat predictions of a neural net as subjective opinions and learn t…
Frequently, population studies feature pyramidally-organized data represented using Hierarchical Bayesian Models (HBM) enriched with plates. These models can become prohibitively large in settings such as neuroimaging, where a sample is composed of a functional MRI signal measured on 300 brain locations, across 4 measurement sessions, and 30 subjects, resulting in around 1 million latent parameters.Such high dimensionality hampers the usage of modern, expressive flow-based techniques.To infer parameter posterior distributions in this challenging class of problems, we designed a novel methodolo…
Scalable Vertical Federated Learning via Data Augmentation and Amortized Inference
Centre for Data Science, Queensland University of Technology（昆士兰科技大学数据科学中心）；School of Mathematical Sciences, Queensland University of Technology（昆士兰科技大学数学科学学院）；Euler Institute, Università della Svizzera italiana (USI)（瑞士意大利语区大学欧拉研究所）；Insubria University（因苏布里亚大学）
State-of-the-art spectral energy distribution (SED) analyses use a Bayesian framework to infer the physical properties of galaxies from observed photometry or spectra. They require sampling from a high-dimensional space of SED model parameters and take $>10-100$ CPU hours per galaxy, which renders them practically infeasible for analyzing the $billions$ of galaxies that will be observed by upcoming galaxy surveys ($e.g.$ DESI, PFS, Rubin, Webb, and Roman). In this work, we present an alternative scalable approach to rigorous Bayesian inference using Amortized Neural Posterior Estimation (ANPE)…
While deep neural networks provide good performance for a range of challenging tasks, calibration and uncertainty estimation remain major challenges, especially under distribution shift. In this paper, we propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation, calibration, and out-of-distribution robustness with deep networks. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principl…
University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；University of British Columbia（不列颠哥伦比亚大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Overcoming Selection Bias in Statistical Studies With Amortized Bayesian Inference
University of Bonn（波恩大学）；Institut Pasteur（巴斯德研究所）；University of Freiburg（弗赖堡大学）；LMU University Hospital（慕尼黑大学附属医院）；German Center for Infection Research（德国感染研究中心）；Fraunhofer Institute ITMP（弗劳恩霍夫ITMP研究所）；Max von Pettenkofer Institute（马克斯·冯·佩滕科弗研究所）；LMU Munich（慕尼黑大学）；German Research Center for Environmental Health（德国环境健康研究中心）；Institut Necker-Enfants Malades（内克-儿童疾病研究所）
Department of Chemical Engineering, University of Michigan（美国密歇根大学化学工程系）；Michigan Institute for Data & AI in Society, University of Michigan（美国密歇根大学数据与社会人工智能研究所）；Department of Statistics, University of Michigan（美国密歇根大学统计系）
Amortized Simulation-Based Inference in Generalized Bayes via Neural Posterior Estimation
Department of Statistics, University of Oxford, Oxford, United Kingdom（英国牛津大学统计系）；Department of Statistics, University of Auckland, Auckland, New Zealand（新西兰奥克兰大学统计系）
