Estimating and disentangling epistemic uncertainty, uncertainty that is reducible with more training data, and aleatoric uncertainty, uncertainty that is inherent to the task at hand, is critically important when applying machine learning to high-stakes applications such as medical imaging and weather forecasting. Conditional diffusion models' breakthrough ability to accurately and efficiently sample from the posterior distribution of a dataset now makes uncertainty estimation conceptually straightforward: One need only train and sample from a large ensemble of diffusion models. Unfortunately, training such an ensemble becomes computationally intractable as the complexity of the model architecture grows. In this work we introduce a new approach to ensembling, hyper-diffusion models (HyperDM), which allows one to accurately estimate both epistemic and aleatoric uncertainty with a single model. Unlike existing single-model uncertainty methods like Monte-Carlo dropout and Bayesian neural networks, HyperDM offers prediction accuracy on par with, and in some cases superior to, multi-model ensembles. Furthermore, our proposed approach scales to modern network architectures such as Attention U-Net and yields more accurate uncertainty estimates compared to existing methods. We validate our method on two distinct real-world tasks: x-ray computed tomography reconstruction and weather temperature forecasting.
HybridFlow: Quantification of Aleatoric and Epistemic Uncertainty with a Single Hybrid Model
Department of Earth, Environment, and Planetary Sciences, Brown University（布朗大学地球、环境与行星科学系）；Data Science Institute, Brown University（布朗大学数据科学研究院）；Department of Computer Science, Brown University（布朗大学计算机科学系）
Age-related macular degeneration (AMD) is one of the leading causes of permanent vision loss in people aged over 60 years. Accurate segmentation of biomarkers such as drusen that points to the early stages of AMD is crucial in preventing further vision impairment. However, segmenting drusen is extremely challenging due to their varied sizes and appearances, low contrast and noise resemblance. Most existing literature, therefore, have focused on size estimation of drusen using classification, leaving the challenge of accurate segmentation less tackled. Additionally, obtaining the pixel-wise ann…
Hamburg University of Technology（汉堡工业大学）；Leibniz Universität Hannover（莱布尼茨汉诺威大学）；Hannover Medical School（汉诺威医学院）；The Hospital for Sick Children（多伦多病童医院）；University of Toronto Mississauga（多伦多大学密西沙加校区）
The consideration of predictive uncertainty in medical imaging with deep learning is of utmost importance. We apply estimation of both aleatoric and epistemic uncertainty by variational Bayesian inference with Monte Carlo dropout to regression tasks and show that predictive uncertainty is systematically underestimated. We apply $ σ$ scaling with a single scalar value; a simple, yet effective calibration method for both types of uncertainty. The performance of our approach is evaluated on a variety of common medical regression data sets using different state-of-the-art convolutional network arc…
University of Groningen（格罗宁根大学）；Pontifical Catholic University of Peru（秘鲁天主大学）
Neural networks are ubiquitous in many tasks, but trusting their predictions is an open issue. Uncertainty quantification is required for many applications, and disentangled aleatoric and epistemic uncertainties are best. In this paper, we generalize methods to produce disentangled uncertainties to work with different uncertainty quantification methods, and evaluate their capability to produce disentangled uncertainties. Our results show that: there is an interaction between learning aleatoric and epistemic uncertainty, which is unexpected and violates assumptions on aleatoric uncertainty, som…
University College London（伦敦大学学院）；King’s College London（伦敦国王学院）；University of Electronic Science and Technology of China（电子科技大学）；University Hospitals Leuven（鲁汶大学医院）；Institute for Women’s Health（妇女健康研究所）
Despite the state-of-the-art performance for medical image segmentation, deep convolutional neural networks (CNNs) have rarely provided uncertainty estimations regarding their segmentation outputs, e.g., model (epistemic) and image-based (aleatoric) uncertainties. In this work, we analyze these different types of uncertainties for CNN-based 2D and 3D medical image segmentation tasks. We additionally propose a test-time augmentation-based aleatoric uncertainty to analyze the effect of different transformations of the input image on the segmentation output. Test-time augmentation has been previo…
A Ground-Truth Framework for Uncertainty Disentanglement with Posterior Risk
University of Tübingen（蒂宾根大学）；Hertie Institute for AI in Brain Health（赫蒂脑健康人工智能研究所）；Tübingen AI Center（蒂宾根人工智能中心）；Charité–Universitätsmedizin Berlin（柏林夏里特医学院）；Bernstein Center for Computational Neuroscience Berlin（柏林伯恩斯坦计算神经科学中心）
Technical University of Munich（慕尼黑工业大学）；Munich Data Science Institute（慕尼黑数据科学研究所）；Stanford University（斯坦福大学）；Stanford Intelligent Systems Laboratory（斯坦福智能系统实验室）
Characterizing aleatoric and epistemic uncertainty on the predicted rewards can help in building reliable reinforcement learning (RL) systems. Aleatoric uncertainty results from the irreducible environment stochasticity leading to inherently risky states and actions. Epistemic uncertainty results from the limited information accumulated during learning to make informed decisions. Characterizing aleatoric and epistemic uncertainty can be used to speed up learning in a training environment, improve generalization to similar testing environments, and flag unfamiliar behavior in anomalous testing …
Graz University of Technology（格拉茨技术大学）；Silicon Austria Labs（奥地利硅谷实验室）；University of Bonn（波恩大学）；Johannes Kepler University（约翰·开普勒大学）；Technical University of Munich（慕尼黑工业大学）；Imperial College London（伦敦帝国理工学院）；NYU School of Medicine（纽约大学医学院）
Recent deep learning approaches focus on improving quantitative scores of dedicated benchmarks, and therefore only reduce the observation-related (aleatoric) uncertainty. However, the model-immanent (epistemic) uncertainty is less frequently systematically analyzed. In this work, we introduce a Bayesian variational framework to quantify the epistemic uncertainty. To this end, we solve the linear inverse problem of undersampled MRI reconstruction in a variational setting. The associated energy functional is composed of a data fidelity term and the total deep variation (TDV) as a learned paramet…
Gaussian processes are Bayesian non-parametric models used in many areas. In this work, we propose a Non-stationary Heteroscedastic Gaussian process model which can be learned with gradient-based techniques. We demonstrate the interpretability of the proposed model by separating the overall uncertainty into aleatoric (irreducible) and epistemic (model) uncertainty. We illustrate the usability of derived epistemic uncertainty on active learning problems. We demonstrate the efficacy of our model with various ablations on multiple datasets.
Due to the steadily increasing relevance of machine learning for practical applications, many of which are coming with safety requirements, the notion of uncertainty has received increasing attention in machine learning research in the last couple of years. In particular, the idea of distinguishing between two important types of uncertainty, often refereed to as aleatoric and epistemic, has recently been studied in the setting of supervised learning. In this paper, we propose to quantify these uncertainties with random forests. More specifically, we show how two general approaches for measurin…
University of Colorado Boulder（科罗拉多大学博尔德分校）；The Charles Stark Draper Laboratory, Inc.（查尔斯·斯塔克·德雷珀实验室公司）
Giving autonomous agents the ability to forecast their own outcomes and uncertainty will allow them to communicate their competencies and be used more safely. We accomplish this by using a learned world model of the agent system to forecast full agent trajectories over long time horizons. Real world systems involve significant sources of both aleatoric and epistemic uncertainty that compound and interact over time in the trajectory forecasts. We develop a deep generative world model that quantifies aleatoric uncertainty while incorporating the effects of epistemic uncertainty during the learni…
CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk
Department of Operations, HEC, University of Lausanne（洛桑大学运营管理系）；Department of Statistics, University of California, Berkeley（加州大学伯克利分校统计系）；Department of Electrical Engineering and Computer Science, University of California, Berkeley（加州大学伯克利分校电气工程与计算机科学系）；BegooAI
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Bristol（布里斯托大学）
Typical Bayesian approaches to OOD detection use epistemic uncertainty. Surprisingly from the Bayesian perspective, there are a number of methods that successfully use aleatoric uncertainty to detect OOD points (e.g. Hendryks et al. 2018). In addition, it is difficult to use outlier exposure to improve a Bayesian OOD detection model, as it is not clear whether it is possible or desirable to increase posterior (epistemic) uncertainty at outlier points. We show that a generative model of data curation provides a principled account of aleatoric uncertainty for OOD detection. In particular, aleato…
Variational LSTM with Augmented Inputs: Nonlinear Response History Metamodeling with Aleatoric and Epistemic Uncertainty
Department of Civil, Environmental, and Construction Engineering, Texas Tech University（德克萨斯理工大学土木、环境与建筑工程系）；Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute（伦斯勒理工学院土木与环境工程系）；National Wind Institute, Texas Tech University（德克萨斯理工大学国家风能研究所）
Cooperative Variance Estimation and Bayesian Neural Networks for Disentangling Aleatoric and Epistemic Uncertainties
Faculty of Mechanical Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628 CD, The Netherlands（代尔夫特理工大学机械工程学院）；School of Engineering, Brown University, 184 Hope St., Providence, RI 02912, USA（布朗大学工程学院）
The notion of uncertainty is of major importance in machine learning and constitutes a key element of machine learning methodology. In line with the statistical tradition, uncertainty has long been perceived as almost synonymous with standard probability and probabilistic predictions. Yet, due to the steadily increasing relevance of machine learning for practical applications and related issues such as safety requirements, new problems and challenges have recently been identified by machine learning scholars, and these problems may call for new methodological developments. In particular, this …
Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisition
Université de Lorraine, CentraleSupélec CNRS, LORIA, Metz, France（洛林大学、中央超导电子学学院 CNRS、LORIA、法国梅茨）；Université de technologie de Compiègne UMR CNRS 7253 Heudiasyc, France（图卢兹理工学院 UMR CNRS 7253 Heudiasyc、法国）；CNRS, Université de technologie de Compiègne UMR CNRS 7253 Heudiasyc, France（CNRS、图卢兹理工学院 UMR CNRS 7253 Heudiasyc、法国）；University of Ghent Ghent, Belgium（根特大学、比利时根特）
A Posterior-Predictive Variance Decomposition for Epistemic and Aleatoric Uncertainty in Wind Power Forecasting
School of Engineering, Deakin University（德肯大学工程学院）；ARC Training Centre in Energy Technologies for Future Grids, School of Engineering, University of Wollongong（未来电网能源技术培训中心，沃林戈大学工程学院）
Kansas State University（堪萨斯州立大学）；ETH Zurich（苏黎世联邦理工学院）
Graph Neural Networks (GNN) provide a powerful framework that elegantly integrates Graph theory with Machine learning for modeling and analysis of networked data. We consider the problem of quantifying the uncertainty in predictions of GNN stemming from modeling errors and measurement uncertainty. We consider aleatoric uncertainty in the form of probabilistic links and noise in feature vector of nodes, while epistemic uncertainty is incorporated via a probability distribution over the model parameters. We propose a unified approach to treat both sources of uncertainty in a Bayesian framework, …
Evaluating deep learning models for fault diagnosis of a rotating machinery with epistemic and aleatoric uncertainty
Faculty of Engineering and Natural Sciences（工程与自然科学学院）；Department of Information and Communications Engineering（信息与通信工程系）；Department of Management, Economics and Industrial Engineering（管理、经济与工业工程系）
Soil moisture is an important variable that determines floods, vegetation health, agriculture productivity, and land surface feedbacks to the atmosphere, etc. Accurately modeling soil moisture has important implications in both weather and climate models. The recently available satellite-based observations give us a unique opportunity to build data-driven models to predict soil moisture instead of using land surface models, but previously there was no uncertainty estimate. We tested Monte Carlo dropout (MCD) with an aleatoric term for our long short-term memory models for this problem, and ask…
Real-Time Structural Health Monitoring with Bayesian Neural Networks: Distinguishing Aleatoric and Epistemic Uncertainty for Digital Twin Frameworks
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））；Korea Institute of Industrial Technology（韩国产业技术研究院）；Kyungpook National University（庆北国立大学）；Korea University of Technology and Education（韩国技术教育大学）
TRIAGE: Type-Routed Interventions via Aleatoric-Epistemic Gated Estimation in Robotic Manipulation and Adaptive Perception -- Don't Treat All Uncertainty the Same
University of Illinois at Chicago（伊利诺伊大学香槟分校）；Intel Labs（英特尔实验室）
Counterfactual explanations (CEs) are a practical tool for demonstrating why machine learning classifiers make particular decisions. For CEs to be useful, it is important that they are easy for users to interpret. Existing methods for generating interpretable CEs rely on auxiliary generative models, which may not be suitable for complex datasets, and incur engineering overhead. We introduce a simple and fast method for generating interpretable CEs in a white-box setting without an auxiliary model, by using the predictive uncertainty of the classifier. Our experiments show that our proposed alg…
Disentangling Aleatoric and Epistemic Uncertainty in Physics-Informed Neural Networks. Application to Insulation Material Degradation Prognostics
Department of Electronics and Computer Science（电子与计算机科学系）；Mondragon University（蒙得龙大学）；Research & Development（研发）；CARTIF Technology Centre（CARTIF技术中心）；Department of Physical Chemistry（物理化学系）；University of the Basque Country (EHU)（巴斯克国家大学（EHU））；Basque Centre for Applied Mathematics (BCAM)（巴斯克应用数学中心（BCAM））；Department of Computer Science and Artificial Intelligence（计算机科学与人工智能系）
CUPID: A Plug-in Framework for Joint Aleatoric and Epistemic Uncertainty Estimation with a Single Model
Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore（南洋理工大学李科钦医学院，新加坡）；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院，新加坡）；Centre for Medical Technologies & Innovations, National Health Group, Singapore（国家健康集团医学技术与创新中心，新加坡）
Heinz Nixdorf Institute（海因茨·尼克斯多夫研究所）；Paderborn University（帕德博恩大学）；UMR CNRS 7253 Heudiasyc（法国国家科学研究中心联合研究单位7253 Heudiasyc）；Sorbonne Universités（索邦大学）；Université de Technologie de Compiègne（贡比涅技术大学）
Various strategies for active learning have been proposed in the machine learning literature. In uncertainty sampling, which is among the most popular approaches, the active learner sequentially queries the label of those instances for which its current prediction is maximally uncertain. The predictions as well as the measures used to quantify the degree of uncertainty, such as entropy, are almost exclusively of a probabilistic nature. In this paper, we advocate a distinction between two different types of uncertainty, referred to as epistemic and aleatoric, in the context of active learning. …
Uncertainty Estimation and Out-of-Distribution Detection for LiDAR Scene Semantic Segmentation
Institute of Cartography and Geoinformatics, Leibniz University Hannover（汉诺威莱布尼茨大学制图与地理信息研究所）；Institute of Photogrammetry and GeoInformation, Leibniz University Hannover（汉诺威莱布尼茨大学摄影测量与地理信息研究所）
Deep learning techniques have been shown to be extremely effective for various classification and regression problems, but quantifying the uncertainty of their predictions and separating them into the epistemic and aleatoric fractions is still considered challenging. In oil and gas exploration projects, tools consisting of seismic, sonic, magnetic resonance, resistivity, dielectric and/or nuclear sensors are sent downhole through boreholes to probe the earth's rock and fluid properties. The measurements from these tools are used to build reservoir models that are subsequently used for estimati…
Intention Inference Under Execution Noise: Separating Aleatoric and Epistemic Uncertainty in Social Dilemmas
University of Cape Town（开普敦大学）；Neuroscience Institute, University of Cape Town（开普敦大学神经科学研究所）；National Institute for Theoretical & Computational Sciences, Stellenbosch（斯泰伦博斯理论与计算科学国家研究所）；Institut National de la recherche scientifique, Montreal（魁北克国家科学研究院（蒙特利尔））
Conformalized Generative Bayesian Imaging: An Uncertainty Quantification Framework for Computational Imaging
Department of Electrical and Computer Engineering, University of Rochester（电气与计算机工程系，罗切斯特大学）；Goergen Institute for Data Science & AI, University of Rochester（数据科学与人工智能研究院，罗切斯特大学）
