Despite the huge success of deep neural networks (NNs), finding good mechanisms for quantifying their prediction uncertainty is still an open problem. Bayesian neural networks are one of the most popular approaches to uncertainty quantification. On the other hand, it was recently shown that ensembles of NNs, which belong to the class of mixture models, can be used to quantify prediction uncertainty. In this paper, we build upon these two approaches. First, we increase the mixture model's flexibility by replacing the fixed mixing weights by an adaptive, input-dependent distribution (specifying the probability of each component) represented by NNs, and by considering uncountably many mixture components. The resulting class of models can be seen as the continuous counterpart to mixture density networks and is therefore referred to as compound density networks (CDNs). We employ both maximum likelihood and variational Bayesian inference to train CDNs, and empirically show that they yield better uncertainty estimates on out-of-distribution data and are more robust to adversarial examples than the previous approaches.
University of Pennsylvania（宾夕法尼亚大学）；Purdue University（普渡大学）；University of Michigan（密歇根大学）
Deep neural network (DNN) models have achieved state-of-the-art predictive accuracy in a wide range of supervised learning applications. However, accurately quantifying the uncertainty in DNN predictions remains a challenging task. For continuous outcome variables, an even more difficult problem is to estimate the predictive density function, which not only provides a natural quantification of the predictive uncertainty, but also fully captures the random variation in the outcome. In this work, we propose the Bayesian Deep Noise Neural Network (B-DeepNoise), which generalizes standard Bayesian…
Uncertainty Quantification in Probabilistic Machine Learning Models: Theory, Methods, and Insights
Weill Cornell Medicine, Cornell University, New York, NY（韦尔·科恩医学中心，康奈尔大学，纽约，纽约）；Department of Electrical and Computer Engineering, Stony Brook University, Stony Brook, NY（电气与计算机工程系，史泰文斯布鲁克大学，史泰文斯布鲁克，纽约）
ONERA（法国国家航空航天研究院）；Université de Toulouse（图卢兹大学）；Zurich University of Applied Sciences（苏黎世应用科学大学）
Reliable uncertainty quantification on RUL prediction is crucial for informative decision-making in predictive maintenance. In this context, we assess some of the latest developments in the field of uncertainty quantification for prognostics deep learning. This includes the state-of-the-art variational inference algorithms for Bayesian neural networks (BNN) as well as popular alternatives such as Monte Carlo Dropout (MCD), deep ensembles (DE) and heteroscedastic neural networks (HNN). All the inference techniques share the same inception deep learning architecture as a functional model. We per…
Harvard Data Science Initiative（哈佛数据科学倡议）；Harvard University（哈佛大学）
Uncertainty quantification has been a core of the statistical machine learning, but its computational bottleneck has been a serious challenge for both Bayesians and frequentists. We propose a model-based framework in quantifying uncertainty, called predictive-matching Generative Parameter Sampler (GPS). This procedure considers an Uncertainty Quantification (UQ) distribution on the targeted parameter, which matches the corresponding predictive distribution to the observed data. This framework adopts a hierarchical modeling perspective such that each observation is modeled by an individual para…
Multi-level Monte Carlo Dropout for Efficient Uncertainty Quantification
Institute for Mathematical Innovation University of Bath, Bath, UK（数学创新研究所 英国巴斯大学）；Department of Mathematical Sciences University of Bath, Bath, UK（数学科学系 英国巴斯大学）
University of Amsterdam（阿姆斯特丹大学）；Radboud University Medical Center（拉德堡德大学医学中心）
The accurate estimation of predictive uncertainty carries importance in medical scenarios such as lung node segmentation. Unfortunately, most existing works on predictive uncertainty do not return calibrated uncertainty estimates, which could be used in practice. In this work we exploit multi-grader annotation variability as a source of 'groundtruth' aleatoric uncertainty, which can be treated as a target in a supervised learning problem. We combine this groundtruth uncertainty with a Probabilistic U-Net and test on the LIDC-IDRI lung nodule CT dataset and MICCAI2012 prostate MRI dataset. We f…
Robust and Reliable AI for Predictive Quality in Semiconductor Materials Manufacturing with MLOps and Uncertainty Quantification
Merck Group（默克集团）；Versum Materials US LLC（Versum Materials美国分公司）；Merck Electronics KGaA（默克电子（KGaA））；Technical University of Darmstadt（达姆施塔特技术大学）；Otto von Guericke University Magdeburg（奥托·冯·格里克大学马格德堡分校）
University of Notre Dame（圣母大学）；Technical University of Munich（慕尼黑工业大学）；University of Pennsylvania（宾夕法尼亚大学）
Surrogate modeling and uncertainty quantification tasks for PDE systems are most often considered as supervised learning problems where input and output data pairs are used for training. The construction of such emulators is by definition a small data problem which poses challenges to deep learning approaches that have been developed to operate in the big data regime. Even in cases where such models have been shown to have good predictive capability in high dimensions, they fail to address constraints in the data implied by the PDE model. This paper provides a methodology that incorporates the…
Deep neural networks have outperformed existing machine learning models in various molecular applications. In practical applications, it is still difficult to make confident decisions because of the uncertainty in predictions arisen from insufficient quality and quantity of training data. Here, we show that Bayesian neural networks are useful to quantify the uncertainty of molecular property prediction with three numerical experiments. In particular, it enables us to decompose the predictive variance into the model- and data-driven uncertainties, which helps to elucidate the source of errors. …
We are interested in the development of surrogate models for uncertainty quantification and propagation in problems governed by stochastic PDEs using a deep convolutional encoder-decoder network in a similar fashion to approaches considered in deep learning for image-to-image regression tasks. Since normal neural networks are data intensive and cannot provide predictive uncertainty, we propose a Bayesian approach to convolutional neural nets. A recently introduced variational gradient descent algorithm based on Stein's method is scaled to deep convolutional networks to perform approximate Baye…
UCL（伦敦大学学院）；University of Amsterdam（阿姆斯特丹大学）；Foundation IRCCS Carlo Besta Neurological Institute（IRCCS卡洛·贝斯塔神经科学研究所基金会）；University of Nottingham（诺丁汉大学）；University of Oxford（牛津大学）；Microsoft Research Cambridge（微软剑桥研究院）
Deep learning (DL) has shown great potential in medical image enhancement problems, such as super-resolution or image synthesis. However, to date, little consideration has been given to uncertainty quantification over the output image. Here we introduce methods to characterise different components of uncertainty in such problems and demonstrate the ideas using diffusion MRI super-resolution. Specifically, we propose to account for $intrinsic$ uncertainty through a heteroscedastic noise model and for $parameter$ uncertainty through approximate Bayesian inference, and integrate the two to quanti…
Epistemic uncertainty quantification is a crucial part of drawing credible conclusions from predictive models, whether concerned about the prediction at a given point or any downstream evaluation that uses the model as input. When the predictive model is simple and its evaluation differentiable, this task is solved by the delta method, where we propagate the asymptotically-normal uncertainty in the predictive model through the evaluation to compute standard errors and Wald confidence intervals. However, this becomes difficult when the model and/or evaluation becomes more complex. Remedies incl…
We present $Δ$-UQ -- a novel, general-purpose uncertainty estimator using the concept of anchoring in predictive models. Anchoring works by first transforming the input into a tuple consisting of an anchor point drawn from a prior distribution, and a combination of the input sample with the anchor using a pretext encoding scheme. This encoding is such that the original input can be perfectly recovered from the tuple -- regardless of the choice of the anchor. Therefore, any predictive model should be able to predict the target response from the tuple alone (since it implicitly represents the in…
The Delta method is a classical procedure for quantifying epistemic uncertainty in statistical models, but its direct application to deep neural networks is prevented by the large number of parameters $P$. We propose a low cost variant of the Delta method applicable to $L_2$-regularized deep neural networks based on the top $K$ eigenpairs of the Fisher information matrix. We address efficient computation of full-rank approximate eigendecompositions in terms of either the exact inverse Hessian, the inverse outer-products of gradients approximation or the so-called Sandwich estimator. Moreover, …
VERSES AI Research Lab（VERSES AI研究实验室）；School of Engineering and Informatics, University of Sussex（萨塞克斯大学工程与信息学院）；Max Planck Institute of Animal Behavior（马克斯·普朗克动物行为研究所）；Duke University（杜克大学）
Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
Drexel University（德雷塞尔大学）；AWS AI Lab（亚马逊云科技人工智能实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Uncertainty quantification in neural networks gained a lot of attention in the past years. The most popular approaches, Bayesian neural networks (BNNs), Monte Carlo dropout, and deep ensembles have one thing in common: they are all based on some kind of mixture model. While the BNNs build infinite mixture models and are derived via variational inference, the latter two build finite mixtures trained with the maximum likelihood method. In this work we investigate the effect of training an infinite mixture distribution with the maximum likelihood method instead of variational inference. We find t…
Conformal and kNN Predictive Uncertainty Quantification Algorithms in Metric Spaces
Department of Economics and Business, Pompeu Fabra University（巴塞罗那庞培法布拉大学经济与商业系）；ICREA；Barcelona Graduate School of Economics（巴塞罗那经济研究生院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Department of Biostatistics, Harvard University（哈佛大学生物统计学系）
Active operator learning with predictive uncertainty quantification for partial differential equations
Sandia National Laboratories（桑迪亚国家实验室）；Department of Statistics（统计学系）；Data Science, Yale University（数据科学，耶鲁大学）；Environmental Engineering, Yale University（环境工程，耶鲁大学）；Department of Mathematics（数学系）；Department of Mechanical Engineering（机械工程系）
Second-Order Uncertainty Quantification: A Distance-Based Approach
Institute of Informatics, LMU Munich（慕尼黑大学信息学研究所）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；PRECISE Center, University of Pennsylvania（宾夕法尼亚大学PRECISE中心）
Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification
Rational Intelligence Lab, CISPA Helmholtz Center for Information Security Department of Statistics, LMU Munich（理性智能实验室，CISPA海德堡信息安全中心统计学系，慕尼黑大学）；Department of Statistics, LMU Munich（统计学系，慕尼黑大学）；Department of Computer Science, The University of Manchester（计算机科学系，曼彻斯特大学）
Harvard University（哈佛大学）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Bayesian Neural Networks (BNNs) place priors over the parameters in a neural network. Inference in BNNs, however, is difficult; all inference methods for BNNs are approximate. In this work, we empirically compare the quality of predictive uncertainty estimates for 10 common inference methods on both regression and classification tasks. Our experiments demonstrate that commonly used metrics (e.g. test log-likelihood) can be misleading. Our experiments also indicate that inference innovations designed to capture structure in the posterior do not necessarily produce high quality posterior approxi…
Heidelberg University（海德堡大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
We propose a novel method for closed-form predictive distribution modeling with neural nets. In quantifying prediction uncertainty, we build on Evidential Deep Learning, which has been impactful as being both simple to implement and giving closed-form access to predictive uncertainty. We employ it to model aleatoric uncertainty and extend it to account also for epistemic uncertainty by converting it to a Bayesian Neural Net. While extending its uncertainty quantification capabilities, we maintain its analytically accessible predictive distribution model by performing progressive moment matchin…
Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Computer Science Department, Stanford University（斯坦福大学计算机科学系）
With increasing deployment of machine learning systems in various real-world tasks, there is a greater need for accurate quantification of predictive uncertainty. While the common goal in uncertainty quantification (UQ) in machine learning is to approximate the true distribution of the target data, many works in UQ tend to be disjoint in the evaluation metrics utilized, and disparate implementations for each metric lead to numerical results that are not directly comparable across different works. To address this, we introduce Uncertainty Toolbox, an open-source python library that helps to ass…
Comparison of Deterministic and Probabilistic Machine Learning Algorithms for Precise Dimensional Control and Uncertainty Quantification in Additive Manufacturing
Center for Nanoscale Materials, Argonne National Laboratory（阿贡国家实验室纳米尺度材料中心）；Department of Mechanical and Industrial Engineering, University of Illinois, Chicago（伊利诺伊大学芝加哥分校机械与工业工程系）；Shell International Exploration and Production Inc.（壳牌国际勘探与生产公司）
The Best Estimate plus Uncertainty (BEPU) approach for nuclear systems modeling and simulation requires that the prediction uncertainty must be quantified in order to prove that the investigated design stays within acceptance criteria. A rigorous Uncertainty Quantification (UQ) process should simultaneously consider multiple sources of quantifiable uncertainties: (1) parameter uncertainty due to randomness or lack of knowledge; (2) experimental uncertainty due to measurement noise; (3) model uncertainty caused by missing/incomplete physics and numerical approximation errors, and (4) code uncer…
Improving Counterfactual Truthfulness for Molecular Property Prediction through Uncertainty Quantification
Institute of Theoretical Informatics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院理论信息学研究所）；Institute of Nanotechnology, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院纳米技术研究所）
Prognostics or Remaining Useful Life (RUL) Estimation from multi-sensor time series data is useful to enable condition-based maintenance and ensure high operational availability of equipment. We propose a novel deep learning based approach for Prognostics with Uncertainty Quantification that is useful in scenarios where: (i) access to labeled failure data is scarce due to rarity of failures (ii) future operational conditions are unobserved and (iii) inherent noise is present in the sensor readings. All three scenarios mentioned are unavoidable sources of uncertainty in the RUL estimation proce…
Traditional Bayesian approaches for model uncertainty quantification rely on notoriously difficult processes of marginalization over each network parameter to estimate its probability density function (PDF). Our hypothesis is that internal layer outputs of a trained neural network contain all of the information related to both its mapping function (quantified by its weights) as well as the input data distribution. We therefore propose a framework for predictive uncertainty quantification of a trained neural network that explicitly estimates the PDF of its raw prediction space (before activatio…
Ensembling neural networks is an effective way to increase accuracy, and can often match the performance of individual larger models. This observation poses a natural question: given the choice between a deep ensemble and a single neural network with similar accuracy, is one preferable over the other? Recent work suggests that deep ensembles may offer distinct benefits beyond predictive power: namely, uncertainty quantification and robustness to dataset shift. In this work, we demonstrate limitations to these purported benefits, and show that a single (but larger) neural network can replicate …
We propose a novel approach to perform approximate Bayesian inference in complex models such as Bayesian neural networks. The approach is more scalable to large data than Markov Chain Monte Carlo, it embraces more expressive models than Variational Inference, and it does not rely on adversarial training (or density ratio estimation). We adopt the recent approach of constructing two models: (1) a primary model, tasked with performing regression or classification; and (2) a secondary, expressive (e.g. implicit) model that defines an approximate posterior distribution over the parameters of the p…
Decision-Oriented Uncertainty Quantification for Risk Control in Earth System Spatiotemporal Foundation Models
Vanderbilt University（范德堡大学）；City University of New York（纽约城市大学）；Yale University（耶鲁大学）；Wyze Inc.（Wyze公司）；Northeastern University（东北大学）；University of Southern California（南加州大学）；Southwest University（西南大学）
Information-theoretic Quantification of High-order Feature Effects in Classification Problems
Faculty of Technical Sciences, University of Novi Sad, Novi Sad, Serbia（塞尔维亚诺维萨德大学技术科学学院）；Department of Engineering, University of Palermo, Palermo, Italy（意大利巴勒莫大学工程系）
