In this paper we introduce a novel way of estimating prediction uncertainty in deep networks through the use of uncertainty surrogates. These surrogates are features of the penultimate layer of a deep network that are forced to match predefined patterns. The patterns themselves can be, among other possibilities, a known visual symbol. We show how our approach can be used for estimating uncertainty in prediction and out-of-distribution detection. Additionally, the surrogates allow for interpretability of the ability of the deep network to learn and at the same time lend robustness against adversarial attacks. Despite its simplicity, our approach is superior to the state-of-the-art approaches on standard metrics as well as computational efficiency and ease of implementation. A wide range of experiments are performed on standard datasets to prove the efficacy of our approach.
For computational efficiency, surrogate models have been used to emulate mathematical simulators for physical or biological processes. High-speed simulation is crucial for conducting uncertainty quantification (UQ) when the simulation is repeated over many randomly sampled input points (aka, the Monte Carlo method). In some cases, UQ is only feasible with a surrogate model. Recently, Deep Neural Network (DNN) surrogate models have gained popularity for their hard-to-match emulation accuracy. However, it is well-known that DNN is prone to errors when input data are perturbed in particular ways,…
Direct Interval Propagation Methods using Neural-Network Surrogates for Uncertainty Quantification in Physical Systems Surrogate Model
LMSD, Department of Mechanical Engineering, KU Leuven（KU莱顿机械工程系LMSD）；RAM, Department of Mechanical Engineering, KU Leuven（KU莱顿机械工程系RAM）；DistriNet, Department of Computer Science, KU Leuven（KU莱顿计算机科学系DistriNet）；Visual Artificial Intelligence Laboratory, Oxford Brookes University（奥克斯福德布鲁克斯大学视觉人工智能实验室）
Uncertainty Quantification for AI-Driven Crash Simulation Surrogates: A Comparative Study of Monte Carlo Dropout and Deep Ensemble on Open-Source Bumper Beam Benchmark
We are interested in the development of surrogate models for uncertainty quantification and propagation in problems governed by stochastic PDEs using a deep convolutional encoder-decoder network in a similar fashion to approaches considered in deep learning for image-to-image regression tasks. Since normal neural networks are data intensive and cannot provide predictive uncertainty, we propose a Bayesian approach to convolutional neural nets. A recently introduced variational gradient descent algorithm based on Stein's method is scaled to deep convolutional networks to perform approximate Baye…
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Jashore University of Science and Technology（杰索尔科技大学）；Indian Institute of Technology Delhi（印度理工学院德里分校）；University of Bristol（布里斯托大学）；Missouri University of Science and Technology（密苏里科技大学）
Industry 4.0 targets the conversion of the traditional industries into intelligent ones through technological revolution. This revolution is only possible through innovation, optimization, interconnection, and rapid decision-making capability. Numerical models are believed to be the key components of Industry 4.0, facilitating quick decision-making through simulations instead of costly experiments. However, numerical investigation of precise, high-fidelity models for optimization or decision-making is usually time-consuming and computationally expensive. In such instances, data-driven surrogat…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Neural networks are powerful surrogates for numerous forward processes. The inversion of such surrogates is extremely valuable in science and engineering. The most important property of a successful neural inverse method is the performance of its solutions when deployed in the real world, i.e., on the native forward process (and not only the learned surrogate). We propose Autoinverse, a highly automated approach for inverting neural network surrogates. Our main insight is to seek inverse solutions in the vicinity of reliable data which have been sampled form the forward process and used for tr…
College of Engineering, Peking University（北京大学工学院）；School of Environmental Science and Engineering, Southern University of Science and Technology（南方科技大学环境科学与工程学院）；Peng Cheng Laboratory（鹏城实验室）
Subsurface flow problems usually involve some degree of uncertainty. Consequently, uncertainty quantification is commonly necessary for subsurface flow prediction. In this work, we propose a methodology for efficient uncertainty quantification for dynamic subsurface flow with a surrogate constructed by the Theory-guided Neural Network (TgNN). The TgNN here is specially designed for problems with stochastic parameters. In the TgNN, stochastic parameters, time and location comprise the input of the neural network, while the quantity of interest is the output. The neural network is trained with a…
TU Dresden（德累斯顿工业大学）；University of Bristol（布里斯托大学）
Explainability of black-box machine learning models is crucial, in particular when deployed in critical applications such as medicine or autonomous cars. Existing approaches produce explanations for the predictions of models, however, how to assess the quality and reliability of such explanations remains an open question. In this paper we take a step further in order to provide the practitioner with tools to judge the trustworthiness of an explanation. To this end, we produce estimates of the uncertainty of a given explanation by measuring the ordinal consensus amongst a set of diverse bootstr…
We propose a novel \textit{capsule} based deep encoder-decoder model for surrogate modeling and uncertainty quantification of systems in mechanics from sparse data. The proposed framework is developed by adapting Capsule Network (CapsNet) architecture into image-to-image regression encoder-decoder network. Specifically, the aim is to exploit the benefits of CapsNet over convolution neural network (CNN) $-$ retaining pose and position information related to an entity to name a few. The performance of proposed approach is illustrated by solving an elliptic stochastic partial differential equatio…
Indian Institute of Technology Guwahati（古瓦哈提印度理工学院）；Indian Institute of Technology Delhi（德里印度理工学院）
We propose a novel deep learning based surrogate model for solving high-dimensional uncertainty quantification and uncertainty propagation problems. The proposed deep learning architecture is developed by integrating the well-known U-net architecture with the Gaussian Gated Linear Network (GGLN) and referred to as the Gated Linear Network induced U-net or GLU-net. The proposed GLU-net treats the uncertainty propagation problem as an image to image regression and hence, is extremely data efficient. Additionally, it also provides estimates of the predictive uncertainty. The network architecture …
University of Notre Dame（圣母大学）；Technical University of Munich（慕尼黑工业大学）；University of Pennsylvania（宾夕法尼亚大学）
Surrogate modeling and uncertainty quantification tasks for PDE systems are most often considered as supervised learning problems where input and output data pairs are used for training. The construction of such emulators is by definition a small data problem which poses challenges to deep learning approaches that have been developed to operate in the big data regime. Even in cases where such models have been shown to have good predictive capability in high dimensions, they fail to address constraints in the data implied by the PDE model. This paper provides a methodology that incorporates the…
DIFFER – Dutch Institute for Fundamental Energy Research（DIFFER——荷兰基础能源研究所）；Eindhoven University of Technology（埃因霍温理工大学）；MIT Plasma Science and Fusion Center（麻省理工学院等离子体科学与聚变中心）；VTT Technical Research Centre of Finland（芬兰VTT技术研究中心）；United Kingdom Atomic Energy Authority（英国原子能管理局）；Forschungszentrum Jülich GmbH（于利希研究中心有限公司）
Micro-Macro Tensor Neural Surrogates for Uncertainty Quantification in Collisional Plasma
University of Ferrara（费拉拉大学）；Italian Ministry of University and Research（意大利大学与研究部）；Royal Society（皇家学会）；Italian National Institute of High Mathematics（意大利高等数学研究院）
SURGIN: SURrogate-guided Generative INversion for subsurface multiphase flow with quantified uncertainty
Department of Geotechnical Engineering, College of Civil Engineering, Tongji University（同济大学土木工程学院地质工程系）；Key Laboratory of Geotechnical and Underground Engineering of Ministry of Education, Tongji University（教育部同济大学地质与地下工程重点实验室）；Physical Science and Engineering Division, King Abdullah University of Science and Technology (KAUST)（卡布斯科学与技术大学（KAUST）物理科学与工程系）；State Key Laboratory of Marine Geology, Tongji University（同济大学海洋地质国家重点实验室）；Tencent SSV（腾讯SSV）；Tencent SD（腾讯SD）
We consider the point evaluation of the solution to interface problems with geometric uncertainties, where the uncertainty in the obstacle is described by a high-dimensional parameter $\boldsymbol{y}\in[-1,1]^d$, $d\in\mathbb{N}$. We focus in particular on an elliptic interface problem and a Helmholtz transmission problem. Point values of the solution in the physical domain depend in general non-smoothly on the high-dimensional parameter, posing a challenge when one is interested in building surrogates. Indeed, high-order methods show poor convergence rates, while methods which are able to tra…
State-of-the-art computer codes for simulating real physical systems are often characterized by a vast number of input parameters. Performing uncertainty quantification (UQ) tasks with Monte Carlo (MC) methods is almost always infeasible because of the need to perform hundreds of thousands or even millions of forward model evaluations in order to obtain convergent statistics. One, thus, tries to construct a cheap-to-evaluate surrogate model to replace the forward model solver. For systems with large numbers of input parameters, one has to deal with the curse of dimensionality - the exponential…
University of Amsterdam（阿姆斯特丹大学）；ITMO University（圣光机大学）
In-Stent Restenosis is a recurrence of coronary artery narrowing due to vascular injury caused by balloon dilation and stent placement. It may lead to the relapse of angina symptoms or to an acute coronary syndrome. An uncertainty quantification of a model for In-Stent Restenosis with four uncertain parameters (endothelium regeneration time, the threshold strain for smooth muscle cells bond breaking, blood flow velocity and the percentage of fenestration in the internal elastic lamina) is presented. Two quantities of interest were studied, namely the average cross-sectional area and the maximu…
National Cheng Kung University（国立成功大学）；Wells Fargo（富国银行）；Georgia Institute of Technology（佐治亚理工学院）
Global optimization of expensive functions has important applications in physical and computer experiments. It is a challenging problem to develop efficient optimization scheme, because each function evaluation can be costly and the derivative information of the function is often not available. We propose a novel global optimization framework using adaptive Radial Basis Functions (RBF) based surrogate model via uncertainty quantification. The framework consists of two iteration steps. It first employs an RBF-based Bayesian surrogate model to approximate the true function, where the parameters …
Uncertainty Quantification for Reduced-Order Surrogate Models Applied to Cloud Microphysics
Department of Applied & Computational Mathematics（应用与计算数学系）；Yale University（耶鲁大学）；Atmospheric, Earth, & Energy Division（大气、地球与能源部门）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Surrogate Modelling of Proton Dose with Monte Carlo Dropout Uncertainty Quantification
Institute for Mathematical Innovation University of Bath, Bath, UK（数学创新研究所 英国巴斯大学）；Department of Mathematical Sciences University of Bath, Bath, UK（数学科学系 英国巴斯大学）
National University of Defense Technology（国防科技大学）；Chinese Academy of Military Science（中国人民解放军军事科学院）
The surrogate model-based uncertainty quantification method has drawn much attention in many engineering fields. Polynomial chaos expansion (PCE) and deep learning (DL) are powerful methods for building a surrogate model. However, PCE needs to increase the expansion order to improve the accuracy of the surrogate model, which causes more labeled data to solve the expansion coefficients, and DL also requires a lot of labeled data to train the deep neural network (DNN). First of all, this paper proposes the adaptive arbitrary polynomial chaos (aPC) and proves two properties about the adaptive exp…
In complex large-scale systems such as climate, important effects are caused by a combination of confounding processes that are not fully observable. The identification of sources from observations of system state is vital for attribution and prediction, which inform critical policy decisions. The difficulty of these types of inverse problems lies in the inability to isolate sources and the cost of simulating computational models. Surrogate models may enable the many-query algorithms required for source identification, but data challenges arise from high dimensionality of the state and source,…
Uncertainty quantification of two-phase flow in porous media via coupled-TgNN surrogate model
China University of Petroleum (East China)（中国石油大学（华东））；Southern University of Science and Technology（南方科技大学）；Peng Cheng Laboratory（鹏城实验室）；Peking University（北京大学）
Scalable Linearized Laplace Approximation via Surrogate Neural Kernel
Machine Learning Group - Computer Science Department Escuela Politécnica Superior（计算机科学系机器学习小组）；Computer Science Department Escuela Politécnica Superior（计算机科学系）；Universidad Autónoma de Madrid（马德里自治大学）；Universidad Pontificia de Comillas（天主教comillas大学）；Aalborg University Copenhagen（哥本哈根奥胡斯大学）；Centro de Investigación Avanzada en Física Fundamental Universidad Autónoma de Madrid（马德里自治大学先进物理基础研究中心）
Peng Cheng Laboratory（鹏城实验室）；Southern University of Science and Technology（南方科技大学）；Peking University（北京大学）
We build surrogate models for dynamic 3D subsurface single-phase flow problems with multiple vertical producing wells. The surrogate model provides efficient pressure estimation of the entire formation at any timestep given a stochastic permeability field, arbitrary well locations and penetration lengths, and a timestep matrix as inputs. The well production rate or bottom hole pressure can then be determined based on Peaceman's formula. The original surrogate modeling task is transformed into an image-to-image regression problem using a convolutional encoder-decoder neural network architecture…
Seismic imaging is an ill-posed inverse problem that is challenged by noisy data and modeling inaccuracies -- due to errors in the background squared-slowness model. Uncertainty quantification is essential for determining how variability in the background models affects seismic imaging. Due to the costs associated with the forward Born modeling operator as well as the high dimensionality of seismic images, quantification of uncertainty is computationally expensive. As such, the main contribution of this work is a survey-specific Fourier neural operator surrogate to velocity continuation that m…
Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）
A Theory-guided Auto-Encoder (TgAE) framework is proposed for surrogate construction and is further used for uncertainty quantification and inverse modeling tasks. The framework is built based on the Auto-Encoder (or Encoder-Decoder) architecture of convolutional neural network (CNN) via a theory-guided training process. In order to achieve the theory-guided training, the governing equations of the studied problems can be discretized and the finite difference scheme of the equations can be embedded into the training of CNN. The residual of the discretized governing equations as well as the dat…
Structure and asymptotic preserving deep neural surrogates for uncertainty quantification in multiscale kinetic equations
University of Ferrara（费拉拉大学）；Italian Ministry of University and Research（意大利大学与研究部）；Royal Society（皇家学会）；European Union（欧盟）；ICSC – Centro Nazionale di Ricerca in High Performance Computing, Big Data and Quantum Computing（ICSC – 高性能计算、大数据和量子计算研究中心）；Italian National Institute of High Mathematics（意大利国家高等数学研究所）
Uncertainty Quantification for Machine Learning-Based Prediction: A Polynomial Chaos Expansion Approach for Joint Model and Input Uncertainty Propagation
School of Mechanical Engineering（机械工程学院）；Purdue University（普渡大学）
Multi-level informed optimization via decomposed Kriging for large design problems under uncertainty
Reliability and Risk Engineering Laboratory, Institute of Energy and Process Engineering, Department of Mechanical and Process Engineering, ETH Zurich（可靠性与风险工程实验室，能源与过程工程研究所，机械与过程工程系，苏黎世联邦理工学院）
When a neural surrogate cannot accelerate a solver: runtime share, closed-loop drift, and the economics of uncertainty gating in a stiff coupled simulation
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Gravitational Physics (Albert Einstein Institute)（马克斯·普朗克引力物理研究所（阿尔伯特·爱因斯坦研究所））
The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modeling
Interdisciplinary Center for Scientific Computing, Heidelberg University, Germany（海德堡大学交叉科学计算中心）；Faculty of Mathematics and Computer Science, Heidelberg University, Germany（海德堡大学数学与计算机科学学院）；Dept. of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Mannheim, Germany（曼海姆中央心理健康研究所理论神经科学部门）；Faculty of Physics and Astronomy, Heidelberg University, Germany（海德堡大学物理与天文学学院）；Hector Institute for AI in Psychiatry and Dept. of Psychiatry and Psychotherapy, CIMH, Mannheim, Germany（曼海姆中央心理健康研究所精神病AI研究所及精神病学与心理学系）；Hertie Institute for AI in Brain Health, University of Tübingen, Germany（图宾根大学脑健康AI研究所）
Aircraft industry is constantly striving for more efficient design optimization methods in terms of human efforts, computation time, and resource consumption. Hybrid surrogate optimization maintains high results quality while providing rapid design assessments when both the surrogate model and the switch mechanism for eventually transitioning to the HF model are calibrated properly. Feedforward neural networks (FNNs) can capture highly nonlinear input-output mappings, yielding efficient surrogates for aircraft performance factors. However, FNNs often fail to generalize over the out-of-distribu…
BMW Group（宝马集团）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Efficient surrogate modelling is a key requirement for uncertainty quantification in data-driven scenarios. In this work, a novel approach of using Sparse Random Features for surrogate modelling in combination with self-supervised dimensionality reduction is described. The method is compared to other methods on synthetic and real data obtained from crashworthiness analyses. The results show a superiority of the here described approach over state of the art surrogate modelling techniques, Polynomial Chaos Expansions and Neural Networks.
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Rutgers University（罗格斯大学）
Uncertainty estimation is an essential step in the evaluation of the robustness for deep learning models in computer vision, especially when applied in risk-sensitive areas. However, most state-of-the-art deep learning models either fail to obtain uncertainty estimation or need significant modification (e.g., formulating a proper Bayesian treatment) to obtain it. Most previous methods are not able to take an arbitrary model off the shelf and generate uncertainty estimation without retraining or redesigning it. To address this gap, we perform a systematic exploration into training-free uncertai…
University of Victoria（维多利亚大学）；Department of Civil Engineering（土木工程系）；Energy and Cities Group（能源与城市小组）
Fast machine learning-based surrogate models are trained to emulate slow, high-fidelity engineering simulation models to accelerate engineering design tasks. This introduces uncertainty as the surrogate is only an approximation of the original model. Bayesian methods can quantify that uncertainty, and deep learning models exist that follow the Bayesian paradigm. These models, namely Bayesian neural networks and Gaussian process models, enable us to give predictions together with an estimate of the model's uncertainty. As a result we can derive uncertainty-aware surrogate models that can automa…
Conditional deep generative models as surrogates for spatial field solution reconstruction with quantified uncertainty in Structural Health Monitoring applications
National Technical University of Athens（雅典国立技术大学）；School of Naval Architecture and Marine Engineering（船舶与海洋工程学院）
Sequential model-based optimization sequentially selects a candidate point by constructing a surrogate model with the history of evaluations, to solve a black-box optimization problem. Gaussian process (GP) regression is a popular choice as a surrogate model, because of its capability of calculating prediction uncertainty analytically. On the other hand, an ensemble of randomized trees is another option and has practical merits over GPs due to its scalability and easiness of handling continuous/discrete mixed variables. In this paper we revisit various ensembles of randomized trees to investig…
Learning Surrogate LPV State-Space Models with Uncertainty Quantification
Eindhoven University of Technology（埃因霍温理工大学）；European Space Agency, ESTEC（欧洲航天局ESTEC）；HUN-REN Institute for Computer Science and Control（HUN-REN计算机科学与控制研究所）
