Bayesian inference and uncertainty quantification in a general class of non-linear inverse regression models is considered. Analytic conditions on the regression model $\{\mathscr G(θ): θ\in Θ\}$ and on Gaussian process priors for $θ$ are provided such that semi-parametrically efficient inference is possible for a large class of linear functionals of $θ$. A general semi-parametric Bernstein-von Mises theorem is proved that shows that the (non-Gaussian) posterior distributions are approximated by certain Gaussian measures centred at the posterior mean. As a consequence posterior-based credible sets are valid and optimal from a frequentist point of view. The theory is illustrated with two applications with PDEs that arise in non-linear tomography problems: an elliptic inverse problem for a Schrödinger equation, and inversion of non-Abelian X-ray transforms. New analytical techniques are deployed to show that the relevant Fisher information operators are invertible between suitable function spaces
This work affords new insights into Bayesian CART in the context of structured wavelet shrinkage. The main thrust is to develop a formal inferential framework for Bayesian tree-based regression. We reframe Bayesian CART as a g-type prior which departs from the typical wavelet product priors by harnessing correlation induced by the tree topology. The practically used Bayesian CART priors are shown to attain adaptive near rate-minimax posterior concentration in the supremum norm in regression models. For the fundamental goal of uncertainty quantification, we construct adaptive confidence bands f…
We begin by introducing the main ideas of the paper under discussion, and we give a brief description of the method proposed. Next, we discuss an alternative approach based on B-spline expansion, and lastly we make some comments on the method's convergence rate.
We propose a new, two-step empirical Bayes-type of approach for neural networks. We show in context of the nonparametric regression model that the procedure (up to a logarithmic factor) provides optimal recovery of the underlying functional parameter of interest and provides Bayesian credible sets with frequentist coverage guarantees. The approach requires fitting the neural network only once, hence it is substantially faster than Bootstrapping type approaches. We demonstrate the applicability of our method over synthetic data, observing good estimation properties and reliable uncertainty quan…
Often the regression function is specified by a system of ordinary differential equations (ODEs) involving some unknown parameters. Typically analytical solution of the ODEs is not available, and hence likelihood evaluation at many parameter values by numerical solution of equations may be computationally prohibitive. Bhaumik and Ghosal (2015) considered a Bayesian two-step approach by embedding the model in a larger nonparametric regression model, where a prior is put through a random series based on B-spline basis functions. A posterior on the parameter is induced from the regression functio…
We provide a general solution to a fundamental open problem in Bayesian inference, namely poor uncertainty quantification, from a frequency standpoint, of Bayesian methods in misspecified models. While existing solutions are based on explicit Gaussian approximations of the posterior, or computationally onerous post-processing procedures, we demonstrate that correct uncertainty quantification can be achieved by replacing the usual posterior with an intuitive approximate posterior. Critically, our solution is applicable to likelihood-based, and generalized, posteriors as well as cases where the …
We investigate the frequentist coverage properties of credible sets resulting in from Gaussian process priors with squared exponential covariance kernel. First we show that by selecting the scaling hyper-parameter using the maximum marginal likelihood estimator in the (slightly modified) squared exponential covariance kernel the corresponding credible sets will provide overconfident, misleading uncertainty statements for a large, representative subclass of the functional parameters in context of the Gaussian white noise model. Then we show that by either blowing up the credible sets with a log…
Deep learning methods continue to have a decided impact on machine learning, both in theory and in practice. Statistical theoretical developments have been mostly concerned with approximability or rates of estimation when recovering infinite dimensional objects (curves or densities). Despite the impressive array of available theoretical results, the literature has been largely silent about uncertainty quantification for deep learning. This paper takes a step forward in this important direction by taking a Bayesian point of view. We study Gaussian approximability of certain aspects of posterior…
Two data-dependent information metrics are developed to quantify the information of the prior and likelihood functions within a parametric Bayesian model, one of which is closely related to the reference priors from Berger, Bernardo, and Sun, and information measure introduced by Lindley. A combination of theoretical, empirical, and computational support provides evidence that these information-theoretic metrics may be useful diagnostic tools when performing a Bayesian analysis.
Pohang University of Science and Technology (POSTECH)（浦项科技大学）
We develop a novel deep learning method for uncertainty quantification in stochastic partial differential equations based on Bayesian neural network (BNN) and Hamiltonian Monte Carlo (HMC). A BNN efficiently learns the posterior distribution of the parameters in deep neural networks by performing Bayesian inference on the network parameters. The posterior distribution is efficiently sampled using HMC to quantify uncertainties in the system. Several numerical examples are shown for both forward and inverse problems in high dimension to demonstrate the effectiveness of the proposed method for un…
We consider the problem of uncertainty quantification for an unknown low-rank matrix $\mathbf{X}$, given a partial and noisy observation of its entries. This quantification of uncertainty is essential for many real-world problems, including image processing, satellite imaging, and seismology, providing a principled framework for validating scientific conclusions and guiding decision-making. However, existing literature has mainly focused on the completion (i.e., point estimation) of the matrix $\mathbf{X}$, with little work on investigating its uncertainty. To this end, we propose in this work…
This work develops rigorous theoretical basis for the fact that deep Bayesian neural network (BNN) is an effective tool for high-dimensional variable selection with rigorous uncertainty quantification. We develop new Bayesian non-parametric theorems to show that a properly configured deep BNN (1) learns the variable importance effectively in high dimensions, and its learning rate can sometimes "break" the curse of dimensionality. (2) BNN's uncertainty quantification for variable importance is rigorous, in the sense that its 95% credible intervals for variable importance indeed covers the truth…
We propose a Bayesian uncertainty quantification method for large-scale imaging inverse problems. Our method applies to all Bayesian models that are log-concave, where maximum-a-posteriori (MAP) estimation is a convex optimization problem. The method is a framework to analyse the confidence in specific structures observed in MAP estimates (e.g., lesions in medical imaging, celestial sources in astronomical imaging), to enable using them as evidence to inform decisions and conclusions. Precisely, following Bayesian decision theory, we seek to assert the structures under scrutiny by performing a…
Geometry-Aware Bayesian Quantification via Compositional Data Analysis
Istituto di Scienza e Tecnologie dell’Informazione Consiglio Nazionale delle Ricerche（国家研究委员会信息科学与技术研究所）；Artificial Intelligence Center University of Oviedo（奥维耶多大学人工智能中心）
We provide a complete framework for performing infinite-dimensional Bayesian inference and uncertainty quantification for image reconstruction with Poisson data. In particular, we address the following issues to make the Bayesian framework applicable in practice. We first introduce a positivity-preserving reparametrization, and we prove that under the reparametrization and a hybrid prior, the posterior distribution is well-posed in the infinite dimensional setting. Second we provide a dimension-independent MCMC algorithm, based on the preconditioned Crank-Nicolson Langevin method, in which we …
We propose a Bayesian statistical model for analyzing coherent anti-Stokes Raman scattering (CARS) spectra. Our quantitative analysis includes statistical estimation of constituent line-shape parameters, underlying Raman signal, error-corrected CARS spectrum, and the measured CARS spectrum. As such, this work enables extensive uncertainty quantification in the context of CARS spectroscopy. Furthermore, we present an unsupervised method for improving spectral resolution of Raman-like spectra requiring little to no \textit{a priori} information. Finally, the recently-proposed wavelet prism metho…
Harvard University（哈佛大学）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Bayesian Neural Networks (BNNs) place priors over the parameters in a neural network. Inference in BNNs, however, is difficult; all inference methods for BNNs are approximate. In this work, we empirically compare the quality of predictive uncertainty estimates for 10 common inference methods on both regression and classification tasks. Our experiments demonstrate that commonly used metrics (e.g. test log-likelihood) can be misleading. Our experiments also indicate that inference innovations designed to capture structure in the posterior do not necessarily produce high quality posterior approxi…
In this work, we propose a two-stage algorithm based on Bayesian modeling and computation aiming at quantifying analyte concentrations or quantities in complex mixtures with Raman spectroscopy. A hierarchical Bayesian model is built for spectral signal analysis, and reversible-jump Markov chain Monte Carlo (RJMCMC) computation is carried out for model selection and spectral variable estimation. Processing is done in two stages. In the first stage, the peak representations for a target analyte spectrum are learned. In the second, the peak variables learned from the first stage are used to estim…
We are interested in the development of surrogate models for uncertainty quantification and propagation in problems governed by stochastic PDEs using a deep convolutional encoder-decoder network in a similar fashion to approaches considered in deep learning for image-to-image regression tasks. Since normal neural networks are data intensive and cannot provide predictive uncertainty, we propose a Bayesian approach to convolutional neural nets. A recently introduced variational gradient descent algorithm based on Stein's method is scaled to deep convolutional networks to perform approximate Baye…
We present a framework that enables estimation of low-dimensional sub-resolution reservoir properties directly from seismic data, without requiring the solution of a high dimensional seismic inverse problem. Our workflow is based on the Bayesian evidential learning approach and exploits learning the direct relation between seismic data and reservoir properties to efficiently estimate reservoir properties. The theoretical framework we develop allows incorporation of non-linear statistical models for seismic estimation problems. Uncertainty quantification is performed with Approximate Bayesian C…
KAIST（韩国科学技术院）；Seoul National University（首尔国立大学）；KAIST Institute for Artificial Intelligence（韩国科学技术院人工智能研究所）
In chemistry, deep neural network models have been increasingly utilized in a variety of applications such as molecular property predictions, novel molecule designs, and planning chemical reactions. Despite the rapid increase in the use of state-of-the-art models and algorithms, deep neural network models often produce poor predictions in real applications because model performance is highly dependent on the quality of training data. In the field of molecular analysis, data are mostly obtained from either complicated chemical experiments or approximate mathematical equations, and then quality …
Optical flow refers to the visual motion observed between two consecutive images. Since the degree of freedom is typically much larger than the constraints imposed by the image observations, the straightforward formulation of optical flow as an inverse problem is ill-posed. Standard approaches to determine optical flow rely on formulating and solving an optimization problem that contains both a data fidelity term and a regularization term, the latter effectively resolves the otherwise ill-posedness of the inverse problem. In this work, we depart from the deterministic formalism, and instead tr…
Uncertainty quantification is essential when dealing with ill-conditioned inverse problems due to the inherent nonuniqueness of the solution. Bayesian approaches allow us to determine how likely an estimation of the unknown parameters is via formulating the posterior distribution. Unfortunately, it is often not possible to formulate a prior distribution that precisely encodes our prior knowledge about the unknown. Furthermore, adherence to handcrafted priors may greatly bias the outcome of the Bayesian analysis. To address this issue, we propose to use the functional form of a randomly initial…
We deal with estimation of multiple dipoles from combined MEG and EEG time--series. We use a sequential Monte Carlo algorithm to characterize the posterior distribution of the number of dipoles and their locations. By considering three test cases, we show that using the combined data the method can localize sources that are not easily (or not at all) visible with either of the two individual data alone. In addition, the posterior distribution from combined data exhibits a lower variance, i.e. lower uncertainty, than the posterior from single device.
We present SPUX - a modular framework for Bayesian inference enabling uncertainty quantification and propagation in linear and nonlinear, deterministic and stochastic models, and supporting Bayesian model selection. SPUX can be coupled to any serial or parallel application written in any programming language, (e.g. including Python, R, Julia, C/C++, Fortran, Java, or a binary executable), scales effortlessly from serial runs on a personal computer to parallel high performance computing clusters, and aims to provide a platform particularly suited to support and foster reproducibility in computa…
Motivation: Epigenetic heterogeneity within a tumour can play an important role in tumour evolution and the emergence of resistance to treatment. It is increasingly recognised that the study of DNA methylation (DNAm) patterns along the genome -- so-called `epialleles' -- offers greater insight into epigenetic dynamics than conventional analyses which examine DNAm marks individually. Results: We have developed a Bayesian model to infer which epialleles are present in multiple regions of the same tumour. We apply our method to reduced representation bisulfite sequencing (RRBS) data from multiple…
What do data tell us about physics-and what don't they tell us? There has been a surge of interest in using machine learning models to discover governing physical laws such as differential equations from data, but current methods lack uncertainty quantification to communicate their credibility. This work addresses this shortcoming from a Bayesian perspective. We introduce a novel model comprising "leaf" modules that learn to represent distinct experiments' spatiotemporal functional data as neural networks and a single "root" module that expresses a nonparametric distribution over their governi…
Despite numerous studies of deep autoencoders (AEs) for unsupervised anomaly detection, AEs still lack a way to express uncertainty in their predictions, crucial for ensuring safe and trustworthy machine learning systems in high-stake applications. Therefore, in this work, the formulation of Bayesian autoencoders (BAEs) is adopted to quantify the total anomaly uncertainty, comprising epistemic and aleatoric uncertainties. To evaluate the quality of uncertainty, we consider the task of classifying anomalies with the additional option of rejecting predictions of high uncertainty. In addition, we…
Due to the importance of uncertainty quantification (UQ), Bayesian approach to inverse problems has recently gained popularity in applied mathematics, physics, and engineering. However, traditional Bayesian inference methods based on Markov Chain Monte Carlo (MCMC) tend to be computationally intensive and inefficient for such high dimensional problems. To address this issue, several methods based on surrogate models have been proposed to speed up the inference process. More specifically, the calibration-emulation-sampling (CES) scheme has been proven to be successful in large dimensional UQ pr…
Motivation: Recent work has demonstrated the feasibility of using non-numerical, qualitative data to parameterize mathematical models. However, uncertainty quantification (UQ) of such parameterized models has remained challenging because of a lack of a statistical interpretation of the objective functions used in optimization. Results: We formulated likelihood functions suitable for performing Bayesian UQ using qualitative data or a combination of qualitative and quantitative data. To demonstrate the resulting UQ capabilities, we analyzed a published model for IgE receptor signaling using synt…
The release of synthetic data generated from a model estimated on the data helps statistical agencies disseminate respondent-level data with high utility and privacy protection. Motivated by the challenge of disseminating sensitive variables containing geographic information in the Consumer Expenditure Surveys (CE) at the U.S. Bureau of Labor Statistics, we propose two non-parametric Bayesian models as data synthesizers for the county identifier of each data record: a Bayesian latent class model and a Bayesian areal model. Both data synthesizers use Dirichlet Process priors to cluster observat…
Local volatility is an important quantity in option pricing, portfolio hedging, and risk management. It is not directly observable from the market; hence calibrations of local volatility models are necessary using observable market data. Unlike most existing point-estimate methods, we cast the large-scale nonlinear inverse problem into the Bayesian framework, yielding a posterior distribution of the local volatility, which naturally quantifies its uncertainty. This extra uncertainty information enables traders and risk managers to make better decisions. To alleviate the computational cost, we …
University of Pennsylvania（宾夕法尼亚大学）；Purdue University（普渡大学）；University of Michigan（密歇根大学）
Deep neural network (DNN) models have achieved state-of-the-art predictive accuracy in a wide range of supervised learning applications. However, accurately quantifying the uncertainty in DNN predictions remains a challenging task. For continuous outcome variables, an even more difficult problem is to estimate the predictive density function, which not only provides a natural quantification of the predictive uncertainty, but also fully captures the random variation in the outcome. In this work, we propose the Bayesian Deep Noise Neural Network (B-DeepNoise), which generalizes standard Bayesian…
Deep neural networks have outperformed existing machine learning models in various molecular applications. In practical applications, it is still difficult to make confident decisions because of the uncertainty in predictions arisen from insufficient quality and quantity of training data. Here, we show that Bayesian neural networks are useful to quantify the uncertainty of molecular property prediction with three numerical experiments. In particular, it enables us to decompose the predictive variance into the model- and data-driven uncertainties, which helps to elucidate the source of errors. …
In the presence of modeling errors, the mainstream Bayesian methods seldom give a realistic account of uncertainties as they commonly underestimate the inherent variability of parameters. This problem is not due to any misconception in the Bayesian framework since it is absolutely robust with respect to the modeling assumptions and the observed data. Rather, this issue has deep roots in users' inability to develop an appropriate class of probabilistic models. This paper bridges this significant gap, introducing a novel Bayesian hierarchical setting, which breaks time-history vibrational respon…
Department of Biostatistics, University of Oslo（奥斯陆大学生物统计系）；Helsinki Institute for Information Technology HIIT, Department of Computer Science, Aalto University（赫尔辛基信息技术研究所，阿尔托大学计算机科学系）
The computational efficiency of approximate Bayesian computation (ABC) has been improved by using surrogate models such as Gaussian processes (GP). In one such promising framework the discrepancy between the simulated and observed data is modelled with a GP which is further used to form a model-based estimator for the intractable posterior. In this article we improve this approach in several ways. We develop batch-sequential Bayesian experimental design strategies to parallellise the expensive simulations. In earlier work only sequential strategies have been used. Current surrogate-based ABC m…
Center for Pattern Recognition and Data Analytics (PRaDA)（模式识别与数据分析中心（PRaDA））；Deakin University（迪肯大学）
In this paper we consider the problem of finding stable maxima of expensive (to evaluate) functions. We are motivated by the optimisation of physical and industrial processes where, for some input ranges, small and unavoidable variations in inputs lead to unacceptably large variation in outputs. Our approach uses multiple gradient Gaussian Process models to estimate the probability that worst-case output variation for specified input perturbation exceeded the desired maxima, and these probabilities are then used to (a) guide the optimisation process toward solutions satisfying our stability cr…
In nuclear reactor system design and safety analysis, the Best Estimate plus Uncertainty (BEPU) methodology requires that computer model output uncertainties must be quantified in order to prove that the investigated design stays within acceptance criteria. "Expert opinion" and "user self-evaluation" have been widely used to specify computer model input uncertainties in previous uncertainty, sensitivity and validation studies. Inverse Uncertainty Quantification (UQ) is the process to inversely quantify input uncertainties based on experimental data in order to more precisely quantify such ad-h…
