IB-UQ: Information bottleneck based uncertainty quantification for neural function regression and neural operator learning
Shanghai Normal University（上海师范大学）；Shanghai Jiaotong University（上海交通大学）；Tongji University（同济大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
We propose a novel framework for uncertainty quantification via information bottleneck (IB-UQ) for scientific machine learning tasks, including deep neural network (DNN) regression and neural operator learning (DeepONet). Specifically, we incorporate the bottleneck by a confidence-aware encoder, which encodes inputs into latent representations according to the confidence of the input data belonging to the region where training data is located, and utilize a Gaussian decoder to predict means and variances of outputs conditional on representation variables. Furthermore, we propose a data augmentation based information bottleneck objective which can enhance the quantification quality of the extrapolation uncertainty, and the encoder and decoder can be both trained by minimizing a tractable variational bound of the objective. In comparison to uncertainty quantification (UQ) methods for scientific learning tasks that rely on Bayesian neural networks with Hamiltonian Monte Carlo posterior estimators, the model we propose is computationally efficient, particularly when dealing with large-scale data sets. The effectiveness of the IB-UQ model has been demonstrated through several representative examples, such as regression for discontinuous functions, real-world data set regression, learning nonlinear operators for partial differential equations, and a large-scale climate model. The experimental results indicate that the IB-UQ model can handle noisy data, generate robust predictions, and provide confident uncertainty evaluation for out-of-distribution data.
We propose a framework for the assessment of uncertainty quantification in deep regression. The framework is based on regression problems where the regression function is a linear combination of nonlinear functions. Basically, any level of complexity can be realized through the choice of the nonlinear functions and the dimensionality of their domain. Results of an uncertainty quantification for deep regression are compared against those obtained by a statistical reference method. The reference method utilizes knowledge of the underlying nonlinear functions and is based on a Bayesian linear reg…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Fabrication process variations can significantly influence the performance and yield of nano-scale electronic and photonic circuits. Stochastic spectral methods have achieved great success in quantifying the impact of process variations, but they suffer from the curse of dimensionality. Recently, low-rank tensor methods have been developed to mitigate this issue, but two fundamental challenges remain open: how to automatically determine the tensor rank and how to adaptively pick the informative simulation samples. This paper proposes a novel tensor regression method to address these two challe…
Uncertainty Quantification for Regression using Proper Scoring Rules
Department of Machine Learning, MBZUAI（机器学习部门，MBZUAI）；ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz（林兹ELLIS单元和LIT AI实验室，机器学习研究所，林兹约翰内斯·开普勒大学）
The ever-growing size of the datasets renders well-studied learning techniques, such as Kernel Ridge Regression, inapplicable, posing a serious computational challenge. Divide-and-conquer is a common remedy, suggesting to split the dataset into disjoint partitions, obtain the local estimates and average them, it allows to scale-up an otherwise ineffective base approach. In the current study we suggest a fully data-driven approach to quantify uncertainty of the averaged estimator. Namely, we construct simultaneous element-wise confidence bands for the predictions yielded by the averaged estimat…
Paris-Saclay University（巴黎萨克雷大学）；ENSTA Paris（巴黎高等先进技术学院）；Institut Polytechnique de Paris（巴黎理工学院）
Monocular depth is important in many tasks, such as 3D reconstruction and autonomous driving. Deep learning based models achieve state-of-the-art performance in this field. A set of novel approaches for estimating monocular depth consists of transforming the regression task into a classification one. However, there is a lack of detailed descriptions and comparisons for Classification Approaches for Regression (CAR) in the community and no in-depth exploration of their potential for uncertainty estimation. To this end, this paper will introduce a taxonomy and summary of CAR approaches, a new un…
Uncertainty Quantification for Regression: A Unified Framework based on kernel scores
LMU Munich（慕尼黑莱茵恩大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；DLR-German Aerospace Center University of Tromsø（德国航空航天中心-特罗姆瑟大学）；German Research Center for Artificial Intelligence (DFKI, DSA)（德国人工智能研究中心）
An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression
LMU Munich（慕尼黑莱布尼茨大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；DLR-German Aerospace Center University of Tromsø（德国航空航天中心图勒大学）；German Research Center for Artificial Intelligence (DFKI, DSA)（德国人工智能研究中心）
While several methods for predicting uncertainty on deep networks have been recently proposed, they do not readily translate to large and complex datasets. In this paper we utilize a simplified form of the Mixture Density Networks (MDNs) to produce a one-shot approach to quantify uncertainty in regression problems. We show that our uncertainty bounds are on-par or better than other reported existing methods. When applied to standard regression benchmark datasets, we show an improvement in predictive log-likelihood and root-mean-square-error when compared to existing state-of-the-art methods. W…
University of California, Berkeley（加州大学伯克利分校）；Technion - Israel Institute of Technology（以色列理工学院）
Image-to-image regression is an important learning task, used frequently in biological imaging. Current algorithms, however, do not generally offer statistical guarantees that protect against a model's mistakes and hallucinations. To address this, we develop uncertainty quantification techniques with rigorous statistical guarantees for image-to-image regression problems. In particular, we show how to derive uncertainty intervals around each pixel that are guaranteed to contain the true value with a user-specified confidence probability. Our methods work in conjunction with any base machine lea…
Increasingly high-stakes decisions are made using neural networks in order to make predictions. Specifically, meteorologists and hedge funds apply these techniques to time series data. When it comes to prediction, there are certain limitations for machine learning models (such as lack of expressiveness, vulnerability of domain shifts and overconfidence) which can be solved using uncertainty estimation. There is a set of expectations regarding how uncertainty should ``behave". For instance, a wider prediction horizon should lead to more uncertainty or the model's confidence should be proportion…
University of Pennsylvania（宾夕法尼亚大学）；Purdue University（普渡大学）；University of Michigan（密歇根大学）
Deep neural network (DNN) models have achieved state-of-the-art predictive accuracy in a wide range of supervised learning applications. However, accurately quantifying the uncertainty in DNN predictions remains a challenging task. For continuous outcome variables, an even more difficult problem is to estimate the predictive density function, which not only provides a natural quantification of the predictive uncertainty, but also fully captures the random variation in the outcome. In this work, we propose the Bayesian Deep Noise Neural Network (B-DeepNoise), which generalizes standard Bayesian…
Lehigh University（理海大学）；Sandia National Laboratories（桑迪亚国家实验室）
We consider the problem of modeling heterogeneous materials where micro-scale dynamics and interactions affect global behavior. In the presence of heterogeneities in material microstructure it is often impractical, if not impossible, to provide quantitative characterization of material response. The goal of this work is to develop a Bayesian framework for uncertainty quantification (UQ) in material response prediction when using nonlocal models. Our approach combines the nonlocal operator regression (NOR) technique and Bayesian inference. Specifically, we use a Markov chain Monte Carlo (MCMC) …
Isotonic Quantile Regression Averaging for uncertainty quantification of electricity price forecasts
Doctoral School, Faculty of Management, Wrocław University of Science and Technology（沃斯克拉大学管理学院博士学院）；Department of Operations Research and Business Intelligence, Wrocław University of Science and Technology（沃斯克拉大学运筹学与商业智能系）
Among the many ways of quantifying uncertainty in a regression setting, specifying the full quantile function is attractive, as quantiles are amenable to interpretation and evaluation. A model that predicts the true conditional quantiles for each input, at all quantile levels, presents a correct and efficient representation of the underlying uncertainty. To achieve this, many current quantile-based methods focus on optimizing the so-called pinball loss. However, this loss restricts the scope of applicable regression models, limits the ability to target many desirable properties (e.g. calibrati…
Enhancing Uncertainty Quantification in Drug Discovery with Censored Regression Labels
AstraZeneca R&D（阿斯利康研发部）；ELLIS Unit Linz（林茨ELLIS单元）；Institute for Machine Learning（机器学习研究所）；Johannes Kepler University Linz（约翰内斯·开普勒大学林茨分校）；KU Leuven（鲁汶大学）；Chalmers University of Technology（查尔姆斯理工大学）
University of Pennsylvania（宾夕法尼亚大学）；Medical College of Wisconsin（威斯康星医学院）；Arizona State University（亚利桑那州立大学）
Bayesian Additive Regression Trees (BART) is a fully Bayesian approach to modeling with ensembles of trees. BART can uncover complex regression functions with high dimensional regressors in a fairly automatic way and provide Bayesian quantification of the uncertainty through the posterior. However, BART assumes IID normal errors. This strong parametric assumption can lead to misleading inference and uncertainty quantification. In this paper, we use the classic Dirichlet process mixture (DPM) mechanism to nonparametrically model the error distribution. A key strength of BART is that default pri…
Model-Free Kernel Conformal Depth Measures Algorithm for Uncertainty Quantification in Regression Models in Separable Hilbert Spaces
Department of Biostatistics（生物统计学系）；Harvard University（哈佛大学）；Department of Epidemiology and Biostatistics（流行病学与生物统计学系）；University of South Carolina（南卡罗来纳大学）；LTCI Telecom Paris Institut Polytechnique de Paris（巴黎电信理工学院LTCI）；UCLA（加州大学洛杉矶分校）
Harvard Data Science Initiative（哈佛数据科学倡议）；Harvard University（哈佛大学）
Uncertainty quantification has been a core of the statistical machine learning, but its computational bottleneck has been a serious challenge for both Bayesians and frequentists. We propose a model-based framework in quantifying uncertainty, called predictive-matching Generative Parameter Sampler (GPS). This procedure considers an Uncertainty Quantification (UQ) distribution on the targeted parameter, which matches the corresponding predictive distribution to the observed data. This framework adopts a hierarchical modeling perspective such that each observation is modeled by an individual para…
University of Michigan（密歇根大学）；Catalysis Science and Technology Institute, University of Michigan（密歇根大学催化科学与技术研究所）
It is critical that machine learning (ML) model predictions be trustworthy for high-throughput catalyst discovery approaches. Uncertainty quantification (UQ) methods allow estimation of the trustworthiness of an ML model, but these methods have not been well explored in the field of heterogeneous catalysis. Herein, we investigate different UQ methods applied to a crystal graph convolutional neural network (CGCNN) to predict adsorption energies of molecules on alloys from the Open Catalyst 2020 (OC20) dataset, the largest existing heterogeneous catalyst dataset. We apply three UQ methods to the…
We present a regression technique for data-driven problems based on polynomial chaos expansion (PCE). PCE is a popular technique in the field of uncertainty quantification (UQ), where it is typically used to replace a runnable but expensive computational model subject to random inputs with an inexpensive-to-evaluate polynomial function. The metamodel obtained enables a reliable estimation of the statistics of the output, provided that a suitable probabilistic model of the input is available. Machine learning (ML) regression is a research field that focuses on providing purely data-driven input…
Cauchy-Schwarz Divergence Information Bottleneck for Regression
Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；UiT - The Arctic University of Norway（挪威北极大学）；Brookhaven National Laboratory（布鲁克海文国家实验室）；NORCE Norwegian Research Centre（NORCE挪威研究中心）；University of Copenhagen（哥本哈根大学）；University of Florida（佛罗里达大学）
E-QRGMM: Efficient Generative Metamodeling for Covariate-Dependent Uncertainty Quantification
School of Data Science（数据科学学院）；School of Management（管理学院）；Department of Decision Analytics and Operations, College of Business, City University of Hong Kong（商业学院决策分析与运营系，香港城市大学）；The Laboratory for AI-Powered Financial Technologies（人工智能赋能金融技术实验室）
Harvard Medical School（哈佛医学院）；Google Research（谷歌研究院）；Harvard School of Public Health（哈佛公共卫生学院）
Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model's uncertainty is evaluated using point-prediction metrics such as negative log-likelihood or the Brier score on heldout data. In this study, we provide the first large scale evaluation of the empirical frequentist coverage properties of well known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage p…
Uncertainty quantification for improving radiomic-based models in radiation pneumonitis prediction
Chakri Naruebodindra Medical Institute, Faculty of Medicine, Ramathibodi Hospital, Mahidol University（玛希隆大学拉玛提博迪医院医学院查克里·纳鲁厄博丁德拉医学研究所）；Department of Clinical Epidemiology and Biostatistics, Faculty of Medicine, Ramathibodi Hospital, Mahidol University（玛希隆大学拉玛提博迪医院医学院临床流行病学与生物统计学系）；Department of Diagnostic and Therapeutic Radiology, Faculty of Medicine, Ramathibodi Hospital, Mahidol University（玛希隆大学拉玛提博迪医院医学院诊断与治疗放射学系）
We are interested in the development of surrogate models for uncertainty quantification and propagation in problems governed by stochastic PDEs using a deep convolutional encoder-decoder network in a similar fashion to approaches considered in deep learning for image-to-image regression tasks. Since normal neural networks are data intensive and cannot provide predictive uncertainty, we propose a Bayesian approach to convolutional neural nets. A recently introduced variational gradient descent algorithm based on Stein's method is scaled to deep convolutional networks to perform approximate Baye…
University of Wuppertal（伍珀塔尔大学）；School of Mathematics and Natural Sciences（数学与自然科学学院）
In object detection with deep neural networks, the box-wise objectness score tends to be overconfident, sometimes even indicating high confidence in presence of inaccurate predictions. Hence, the reliability of the prediction and therefore reliable uncertainties are of highest interest. In this work, we present a post processing method that for any given neural network provides predictive uncertainty estimates and quality estimates. These estimates are learned by a post processing model that receives as input a hand-crafted set of transparent metrics in form of a structured dataset. Therefrom,…
Harvard University（哈佛大学）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Bayesian Neural Networks (BNNs) place priors over the parameters in a neural network. Inference in BNNs, however, is difficult; all inference methods for BNNs are approximate. In this work, we empirically compare the quality of predictive uncertainty estimates for 10 common inference methods on both regression and classification tasks. Our experiments demonstrate that commonly used metrics (e.g. test log-likelihood) can be misleading. Our experiments also indicate that inference innovations designed to capture structure in the posterior do not necessarily produce high quality posterior approxi…
Dynamic mode decomposition (DMD) provides a regression framework for adaptively learning a best-fit linear dynamics model over snapshots of temporal, or spatio-temporal, data. A diversity of regression techniques have been developed for producing the linear model approximation whose solutions are exponentials in time. For spatio-temporal data, DMD provides low-rank and interpretable models in the form of dominant modal structures along with their exponential/oscillatory behavior in time. The majority of DMD algorithms, however, are prone to bias errors from noisy measurements of the dynamics, …
On the Uncertainty Quantification Ability of Tabular Foundation Models
Department of Mechanical and Aerospace Engineering, University of California, Irvine（加州大学欧文分校机械与航空航天工程系）；Department of Civil and Environmental Engineering, University of California, Irvine（加州大学欧文分校土木与环境工程系）
Evidential time-to-event prediction with calibrated uncertainty quantification
Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学苏瑞福公共卫生学院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；Université de Technologie de Compiègne, CNRS, Heudiasyc（贡比涅技术大学、法国国家科研中心、Heudiasyc实验室）；Institut universitaire de France（法国大学研究院）
