It is critical that machine learning (ML) model predictions be trustworthy for high-throughput catalyst discovery approaches. Uncertainty quantification (UQ) methods allow estimation of the trustworthiness of an ML model, but these methods have not been well explored in the field of heterogeneous catalysis. Herein, we investigate different UQ methods applied to a crystal graph convolutional neural network (CGCNN) to predict adsorption energies of molecules on alloys from the Open Catalyst 2020 (OC20) dataset, the largest existing heterogeneous catalyst dataset. We apply three UQ methods to the adsorption energy predictions, namely k-fold ensembling, Monte Carlo dropout, and evidential regression. The effectiveness of each UQ method is assessed based on accuracy, sharpness, dispersion, calibration, and tightness. Evidential regression is demonstrated to be a powerful approach for rapidly obtaining tunable, competitively trustworthy UQ estimates for heterogeneous catalysis applications when using neural networks. Recalibration of model uncertainties is shown to be essential in practical screening applications of catalysts using uncertainties.
Despite numerous studies of deep autoencoders (AEs) for unsupervised anomaly detection, AEs still lack a way to express uncertainty in their predictions, crucial for ensuring safe and trustworthy machine learning systems in high-stake applications. Therefore, in this work, the formulation of Bayesian autoencoders (BAEs) is adopted to quantify the total anomaly uncertainty, comprising epistemic and aleatoric uncertainties. To evaluate the quality of uncertainty, we consider the task of classifying anomalies with the additional option of rejecting predictions of high uncertainty. In addition, we…
Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inserm（法国国家健康与医学研究院）；Pixyl Research and Development Laboratory（Pixyl研发实验室）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）
The full acceptance of Deep Learning (DL) models in the clinical field is rather low with respect to the quantity of high-performing solutions reported in the literature. Particularly, end users are reluctant to rely on the rough predictions of DL models. Uncertainty quantification methods have been proposed in the literature as a potential response to reduce the rough decision provided by the DL black box and thus increase the interpretability and the acceptability of the result by the final user. In this review, we propose an overview of the existing methods to quantify uncertainty associate…
University of Colorado Boulder（科罗拉多大学博尔德分校）；The Charles Stark Draper Laboratory, Inc.（查尔斯·斯塔克·德雷珀实验室公司）
For safe and reliable deployment in the real world, autonomous agents must elicit appropriate levels of trust from human users. One method to build trust is to have agents assess and communicate their own competencies for performing given tasks. Competency depends on the uncertainties affecting the agent, making accurate uncertainty quantification vital for competency assessment. In this work, we show how ensembles of deep generative models can be used to quantify the agent's aleatoric and epistemic uncertainties when forecasting task outcomes as part of competency assessment.
The regulatory approval and broad clinical deployment of medical AI have been hampered by the perception that deep learning models fail in unpredictable and possibly catastrophic ways. A lack of statistically rigorous uncertainty quantification is a significant factor undermining trust in AI results. Recent developments in distribution-free uncertainty quantification present practical solutions for these issues by providing reliability guarantees for black-box models on arbitrary data distributions as formally valid finite-sample prediction intervals. Our work applies these new uncertainty qua…
Uncertainty quantification by ensemble learning is explored in terms of an application from computational optical form measurements. The application requires to solve a large-scale, nonlinear inverse problem. Ensemble learning is used to extend a recently developed deep learning approach for this application in order to provide an uncertainty quantification of its predicted solution to the inverse problem. By systematically inserting out-of-distribution errors as well as noisy data the reliability of the developed uncertainty quantification is explored. Results are encouraging and the proposed…
This paper proposes a new data-driven method for the reliable prediction of power system post-fault trajectories. The proposed method is based on the fundamentally new concept of Deep Operator Networks (DeepONets). Compared to traditional neural networks that learn to approximate functions, DeepONets are designed to approximate nonlinear operators. Under this operator framework, we design a DeepONet to (1) take as inputs the fault-on trajectories collected, for example, via simulation or phasor measurement units, and (2) provide as outputs the predicted post-fault trajectories. In addition, we…
Logit-Level Uncertainty Quantification in Vision-Language Models for Histopathology Image Analysis
Department of Electrical and Electronics Engineering, Izmir Bakircay University（电气与电子工程系，伊兹密尔巴克伊大学）；Department of Electrical Engineering and Computer Science, University of Stavanger（电气工程与计算机科学系，斯塔万格大学）；Batten College of Engineering and Technology, Old Dominion University（工程与技术学院，老奥布良大学）
A biosignal is a signal that can be continuously measured from human bodies, such as respiratory sounds, heart activity (ECG), brain waves (EEG), etc, based on which, machine learning models have been developed with very promising performance for automatic disease detection and health status monitoring. However, dataset shift, i.e., data distribution of inference varies from the distribution of the training, is not uncommon for real biosignal-based applications. To improve the robustness, probabilistic models with uncertainty quantification are adapted to capture how reliable a prediction is. …
Uncertainty quantification is crucial for building reliable and trustable machine learning systems. We propose to estimate uncertainty in recurrent neural networks (RNNs) via stochastic discrete state transitions over recurrent timesteps. The uncertainty of the model can be quantified by running a prediction several times, each time sampling from the recurrent state transition distribution, leading to potentially different results if the model is uncertain. Alongside uncertainty quantification, our proposed method offers several advantages in different settings. The proposed method can (1) lea…
Uncertainty Quantification for AI-Driven Crash Simulation Surrogates: A Comparative Study of Monte Carlo Dropout and Deep Ensemble on Open-Source Bumper Beam Benchmark
Knowing when to trust machine-learned interatomic potentials
Department of Chemistry, Mellon College of Science（化学系，梅隆科学学院）；Department of Materials Science and Engineering（材料科学与工程系）；Ray and Stephanie Lane Computational Biology Department, School of Computer Science（Ray和Stephanie Lane计算生物学部门，计算机科学学院）
Amazon Search（亚马逊搜索）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Uncertainty quantification is one of the most crucial tasks to obtain trustworthy and reliable machine learning models for decision making. However, most research in this domain has only focused on problems with small label spaces and ignored eXtreme Multi-label Classification (XMC), which is an essential task in the era of big data for web-scale machine learning applications. Moreover, enormous label spaces could also lead to noisy retrieval results and intractable computational challenges for uncertainty quantification. In this paper, we aim to investigate general uncertainty quantification …
Trustworthy deployment of ML models requires a proper measure of uncertainty, especially in safety-critical applications. We focus on uncertainty quantification (UQ) for classification problems via two avenues -- prediction sets using conformal prediction and calibration of probabilistic predictors by post-hoc binning -- since these possess distribution-free guarantees for i.i.d. data. Two common ways of generalizing beyond the i.i.d. setting include handling covariate and label shift. Within the context of distribution-free UQ, the former has already received attention, but not the latter. It…
Enhancing Uncertainty Quantification in Drug Discovery with Censored Regression Labels
AstraZeneca R&D（阿斯利康研发部）；ELLIS Unit Linz（林茨ELLIS单元）；Institute for Machine Learning（机器学习研究所）；Johannes Kepler University Linz（约翰内斯·开普勒大学林茨分校）；KU Leuven（鲁汶大学）；Chalmers University of Technology（查尔姆斯理工大学）
Trust-Aware Predictive Emissions Monitoring for Gas Turbine Fleets with Limited Labelled Data
School of Natural and Computing Sciences University of Aberdeen（自然与计算科学学院 布拉德福德大学）；School of Computing Sciences University of East Anglia（计算科学学院 东安格利亚大学）；Siemens Energy Industrial Turbomachinery Ltd.（西门子能源工业涡轮机有限公司）；Department of Physics and Technology UiT The Arctic University of Norway（物理与技术系 UiT 北极大学）
On the Need to Align Intent and Implementation in Uncertainty Quantification for Machine Learning
Fermi National Accelerator Laboratory（费米国家加速器实验室）；Department of Astronomy and Astrophysics, University of Chicago（芝加哥大学天文学与天体物理学系）；Kavli Institute for Cosmological Physics, University of Chicago（芝加哥大学凯弗利宇宙物理研究所）
Uncertainty quantification in fine-tuned LLMs using LoRA ensembles
Stockholm University Department of Physics（斯德哥尔摩大学物理系）；Department of Mathematical Sciences（数学科学系）；Chalmers university of technology & University of Gothenburg（楚德勒斯技术大学及哥德堡大学）；VERSES Research Lab（VERSES研究实验室）
Accurate and trustworthy epidemic forecasting is an important problem that has impact on public health planning and disease mitigation. Most existing epidemic forecasting models disregard uncertainty quantification, resulting in mis-calibrated predictions. Recent works in deep neural models for uncertainty-aware time-series forecasting also have several limitations; e.g. it is difficult to specify meaningful priors in Bayesian NNs, while methods like deep ensembling are computationally expensive in practice. In this paper, we fill this important gap. We model the forecasting task as a probabil…
Metropolis-Scale Resilient and Trustworthy Traffic Flow Inference Using Multi-Source Data
School of Transportation, Jilin University（吉林大学交通运输学院）；Department of Computer Science, City University of Hong Kong (Dongguan)（香港城市大学（东莞）计算机科学系）；Institute for Transport Planning and Systems, ETH Zurich（苏黎世联邦理工学院交通规划与系统研究所）；Institute of Intelligent Transportation Systems, College of Civil Engineering and Architecture, Zhejiang University（浙江大学智能交通系统研究所）；ZJU-UIUC Institute, Zhejiang University（浙江大学-UIUC研究院）
Ensuring trustworthiness in machine learning (ML) models is a multi-dimensional task. In addition to the traditional notion of predictive performance, other notions such as privacy, fairness, robustness to distribution shift, adversarial robustness, interpretability, explainability, and uncertainty quantification are important considerations to evaluate and improve (if deficient). However, these sub-disciplines or 'pillars' of trustworthiness have largely developed independently, which has limited us from understanding their interactions in real-world ML pipelines. In this paper, focusing spec…
Uncertainty Quantification with the Empirical Neural Tangent Kernel
School of Mathematics And Physics University of Queensland（数学与物理学院，昆士兰大学）；Dept. of Electrical and Electronic Engineering University of Melbourne（电子与电气工程系，墨尔本大学）；School of Mathematics and Statistics University of Melbourne（数学与统计学学院，墨尔本大学）；CIRES and School of Mathematics And Physics University of Queensland（CIRES和数学与物理学院，昆士兰大学）
