We consider the problem of calibration and uncertainty analysis for activity-based transportation simulators. Activity-Based Models (ABMs) rely on statistical modeling of individual travelers' behavior to predict higher-order travel patterns in metropolitan areas. Input parameters are typically estimated from traveler surveys using maximum likelihood. We develop an approach that uses a Gaussian Process emulator to calibrate those parameters using traffic flow data. Our approach extends traditional emulators to handle the high-dimensional and non-stationary nature of transportation simulators. We introduce a deep learning dimensionality reduction model that is jointly estimated with Gaussin Process model to approximate the simulator. We demonstrate the methodology using several simulated examples as well as by calibrating key parameters of the Bloomington, Illinois model.
We introduce a Bayesian solution for the problem in forensic speaker recognition, where there may be very little background material for estimating score calibration parameters. We work within the Bayesian paradigm of evidence reporting and develop a principled probabilistic treatment of the problem, which results in a Bayesian likelihood-ratio as the vehicle for reporting weight of evidence. We show in contrast, that reporting a likelihood-ratio distribution does not solve this problem. Our solution is experimentally exercised on a simulated forensic scenario, using NIST SRE'12 scores, which …
Statistical calibration where the curve is nonlinear is important in many areas, such as analytical chemistry and radiometry. Especially in radiometry, instrument characteristics change over time, thus calibration is a process that must be conducted as long as the instrument is in use. We propose a dynamic Bayesian method to perform calibration in the presence of a curvilinear relationship between the reference measurements and the response variable. The dynamic calibration approach adequately derives time dependent calibration distributions in the presence of drifting regression parameters. T…
Experimental calibration of dynamic thermal models is required for model predictive control and characterization of building energy performance. In these applications, the uncertainty assessment of the parameter estimates is decisive; this is why a Bayesian calibration procedure (selection, calibration and validation) is presented. The calibration is based on an improved Metropolis-Hastings algorithm suitable for linear and Gaussian state-space models. The procedure, illustrated on a real house experiment, shows that the algorithm is more robust to initial conditions than a maximum likelihood …
The Mars rover Curiosity carries an instrument called ChemCam to determine the composition of the soil and rocks. ChemCam uses laser-induced breakdown spectroscopy (LIBS) for this purpose. Los Alamos National Laboratory has developed a simulation capability that can predict spectra from ChemCam, but there are major scale differences between the prediction and observation. This presents a challenge when using Bayesian model calibration to determine the unknown physical parameters that describe the LIBS observations. We present an analysis of LIBS data to support ChemCam based on including a str…
We introduce a Bayesian approach to predictive density calibration and combination that accounts for parameter uncertainty and model set incompleteness through the use of random calibration functionals and random combination weights. Building on the work of Ranjan, R. and Gneiting, T. (2010) and Gneiting, T. and Ranjan, R. (2013), we use infinite beta mixtures for the calibration. The proposed Bayesian nonparametric approach takes advantage of the flexibility of Dirichlet process mixtures to achieve any continuous deformation of linearly combined predictive distributions. The inference procedu…
We study statistical calibration, i.e., adjusting features of a computational model that are not observable or controllable in its associated physical system. We focus on functional calibration, which arises in many manufacturing processes where the unobservable features, called calibration variables, are a function of the input variables. A major challenge in many applications is that computational models are expensive and can only be evaluated a limited number of times. Furthermore, without making strong assumptions, the calibration variables are not identifiable. We propose Bayesian non-iso…
We build on the work in Fackler and King 1990, and propose a more general calibration model for implied risk neutral densities. Our model allows for the joint calibration of a set of densities at different maturities and dates through a Bayesian dynamic Beta Markov Random Field. Our approach allows for possible time dependence between densities with the same maturity, and for dependence across maturities at the same point in time. This approach to the problem encompasses model flexibility, parameter parsimony and, more importantly, information pooling across densities.
There are many practical difficulties in the calibration of computer models to experimental data. One such complication is the fact that certain combinations of the calibration inputs can cause the code to output data lacking fundamental properties, or even to produce no output at all. In many cases the researchers want or need to exclude the possibility of these "failures" within their analyses. We propose a Bayesian (meta-)model in which the posterior distribution for the calibration parameters naturally excludes regions of the input space corresponding to failed runs. That is, we define a s…
Universitat Politècnica de València（瓦伦西亚理工大学）；Universidad Autónoma de Madrid（马德里自治大学）
Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy performance in many tasks. However, recent works have pointed out that the outputs provided by these models are not well-calibrated, seriously limiting their use in critical decision scenarios. In this work, we propose to use a decoupled Bayesian stage, implemented with a Bayesian Neural Network (BNN), to map the uncalibrated probabilities provided by a DNN to calibrated ones, consistently improving calibration. Our results evidence that incorporating uncertainty provides more reliable probabilistic models, a critical conditio…
Simulation models of epidemiological, biological, ecological, and environmental processes are increasingly being calibrated using Bayesian statistics. The Bayesian approach provides simple rules to synthesise multiple data sources and to calculate uncertainty in model output due to uncertainty in the calibration data. As the number of tutorials and studies published grow, the solutions to common difficulties in Bayesian calibration across these fields have become more apparent, and a step-by-step process for successful calibration across all these fields is emerging. We provide a statement of …
We find empirical evidence that mean-reverting jump processes are not statistically adequate to model electricity spot price spikes but independent, signed sums of such processes are statistically adequate. Further we demonstrate a change in the composition of these sums after a major economic event. This is achieved by developing a Markov Chain Monte Carlo (MCMC) procedure for Bayesian model calibration and a Bayesian assessment of model adequacy (posterior predictive checking). In particular we determine the number of signed mean-reverting jump components required in the APXUK and EEX market…
This paper presents a calibration method for the speed-flow model of the HCM 2016 for freeways and multilane highways allied to temporal analysis of traffic stream. The proposed method was developed using a sample of more than one million observations collected by 23 traffic sensors on four highways in the state of S{ã}o Paulo. The method is structured on Bayesian inference and provided for each model parameters a probability distribution function. The free-flow speed and capacity presented a probability density function that approximates a Normal distribution. The segment in which the speed o…
With the advancements of computer architectures, the use of computational models proliferates to solve complex problems in many scientific applications such as nuclear physics and climate research. However, the potential of such models is often hindered because they tend to be computationally expensive and consequently ill-fitting for uncertainty quantification. Furthermore, they are usually not calibrated with real-time observations. We develop a computationally efficient algorithm based on variational Bayes inference (VBI) for calibration of computer models with Gaussian processes. Unfortuna…
Probabilistic predictions from neural networks which account for predictive uncertainty during classification is crucial in many real-world and high-impact decision making settings. However, in practice most datasets are trained on non-probabilistic neural networks which by default do not capture this inherent uncertainty. This well-known problem has led to the development of post-hoc calibration procedures, such as Platt scaling (logistic), isotonic and beta calibration, which transforms the scores into well calibrated empirical probabilities. A plausible alternative to the calibration approa…
We develop a Bayesian approach called Bayesian projected calibration to address the problem of calibrating an imperfect computer model using observational data from a complex physical system. The calibration parameter and the physical system are parametrized in an identifiable fashion via $L_2$-projection. The physical process is assigned a Gaussian process prior, which naturally induces a prior distribution on the calibration parameter through the $L_2$-projection constraint. The calibration parameter is estimated through its posterior distribution, which provides a natural and non-asymptotic…
We consider the Bayesian calibration of models describing the phenomenon of block copolymer (BCP) self-assembly using image data produced by microscopy or X-ray scattering techniques. To account for the random long-range disorder in BCP equilibrium structures, we introduce auxiliary variables to represent this aleatory uncertainty. These variables, however, result in an integrated likelihood for high-dimensional image data that is generally intractable to evaluate. We tackle this challenging Bayesian inference problem using a likelihood-free approach based on measure transport together with th…
As we move away from the data, the predictive uncertainty should increase, since a great variety of explanations are consistent with the little available information. We introduce Distance-Aware Prior (DAP) calibration, a method to correct overconfidence of Bayesian deep learning models outside of the training domain. We define DAPs as prior distributions over the model parameters that depend on the inputs through a measure of their distance from the training set. DAP calibration is agnostic to the posterior inference method, and it can be performed as a post-processing step. We demonstrate it…
Romanian Institute of Science and Technology（罗马尼亚科学技术研究院）
Bayesian Neural Networks (BNNs) often result uncalibrated after training, usually tending towards overconfidence. Devising effective calibration methods with low impact in terms of computational complexity is thus of central interest. In this paper we present calibration methods for BNNs based on the alpha divergences from Information Geometry. We compare the use of alpha divergence in training and in calibration, and we show how the use in calibration provides better calibrated uncertainty estimates for specific choices of alpha and is more efficient especially for complex network architectur…
Microsimulation models (MSMs) are used to predict population-level effects of health care policies by simulating individual-level outcomes. Simulated outcomes are governed by unknown parameters that are chosen so that the model accurately predicts specific targets, a process referred to as model calibration. Calibration targets can come from randomized controlled trials, observational studies, and expert opinion, and are typically summary statistics. A well calibrated model can reproduce a wide range of targets. MSM calibration generally involves searching a high dimensional parameter space an…
Molecular dynamics (MD) simulations give access to equilibrium structures and dynamic properties given an ergodic sampling and an accurate force-field. The force-field parameters are calibrated to reproduce properties measured by experiments or simulations. The main contribution of this paper is an approximate Bayesian framework for the calibration and uncertainty quantification of the force-field parameters, without assuming parameter uncertainty to be Gaussian. To this aim, since the likelihood function of the MD simulation models are intractable in absence of Gaussianity assumption, we use …
Bayesian calibration of black-box computer models offers an established framework to obtain a posterior distribution over model parameters. Traditional Bayesian calibration involves the emulation of the computer model and an additive model discrepancy term using Gaussian processes; inference is then carried out using MCMC. These choices pose computational and statistical challenges and limitations, which we overcome by proposing the use of approximate Deep Gaussian processes and variational inference techniques. The result is a practical and scalable framework for calibration, which obtains co…
University of Piuttsburgh（匹兹堡大学）；University of Pittsburgh（匹兹堡大学）
A set of probabilistic predictions is well calibrated if the events that are predicted to occur with probability p do in fact occur about p fraction of the time. Well calibrated predictions are particularly important when machine learning models are used in decision analysis. This paper presents two new non-parametric methods for calibrating outputs of binary classification models: a method based on the Bayes optimal selection and a method based on the Bayesian model averaging. The advantage of these methods is that they are independent of the algorithm used to learn a predictive model, and th…
A common approach to assess the performance of fire insulation panels is the component additive method (CAM). The parameters of the CAM are based on the temperature-dependent thermal material properties of the panels. These material properties can be derived by calibrating finite element heat transfer models using experimentally measured temperature records. In the past, the calibration of the material properties was done manually by trial and error approaches, which was inefficient and prone to error. In this contribution, the calibration problem is reformulated in a probabilistic setting and…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；IBM Research（IBM研究院）
Bayesian decision theory provides an elegant framework for acting optimally under uncertainty when tractable posterior distributions are available. Modern Bayesian models, however, typically involve intractable posteriors that are approximated with, potentially crude, surrogates. This difficulty has engendered loss-calibrated techniques that aim to learn posterior approximations that favor high-utility decisions. In this paper, focusing on Bayesian neural networks, we develop methods for correcting approximate posterior predictive distributions encouraging them to prefer high-utility decisions…
In model development, model calibration and validation play complementary roles toward learning reliable models. In this article, we expand the Bayesian Validation Metric framework to a general calibration and validation framework by inverting the validation mathematics into a generalized Bayesian method for model calibration and regression. We perform Bayesian regression based on a user's definition of model-data agreement. This allows for model selection on any type of data distribution, unlike Bayesian and standard regression techniques, that "fail" in some cases. We show that our tool is c…
This work is motivated by personalized digital twins based on observations and physical models for treatment and prevention of Hypertension. The models commonly used are simplification of the real process and the aim is to make inference about physically interpretable parameters. To account for model discrepancy we propose to set up the estimation problem in a Bayesian calibration framework. This naturally solves the inverse problem accounting for and quantifying the uncertainty in the model formulation, in the parameter estimates and predictions. We focus on the inverse problem, i.e. to estim…
Schervish (1985b) showed that every forecasting system is noncalibrated for uncountably many data sequences that it might see. This result is strengthened here: from a topological point of view, failure of calibration is typical and calibration rare. Meanwhile, Bayesian forecasters are certain that they are calibrated---this invites worries about the connection between Bayesianism and rationality.
In this paper, we aim to design robust estimation techniques based on the compound-Gaussian (CG) process and adapted for calibration of radio interferometers. The motivation beyond this is due to the presence of outliers leading to an unrealistic traditional Gaussian noise assumption. Consequently, to achieve robustness, we adopt a maximum a posteriori (MAP) approach which exploits Bayesian statistics and follows a sequential updating procedure here. The proposed algorithm is applied in a multi-frequency scenario in order to enhance the estimation and correction of perturbation effects. Numeri…
This paper proposes a novel fixed inducing points online Bayesian calibration (FIPO-BC) algorithm to efficiently learn the model parameters using a benchmark database. The standard Bayesian calibration (STD-BC) algorithm provides a statistical method to calibrate the parameters of computationally expensive models. However, the STD-BC algorithm scales very badly with the number of data points and lacks online learning capability. The proposed FIPO-BC algorithm greatly improves the computational efficiency and enables the online calibration by executing the calibration on a set of predefined ind…
Verifying the correctness of Bayesian computation is challenging. This is especially true for complex models that are common in practice, as these require sophisticated model implementations and algorithms. In this paper we introduce \emph{simulation-based calibration} (SBC), a general procedure for validating inferences from Bayesian algorithms capable of generating posterior samples. This procedure not only identifies inaccurate computation and inconsistencies in model implementations but also provides graphical summaries that can indicate the nature of the problems that arise. We argue that…
Purpose: Bayesian calibration is theoretically superior to standard direct-search algorithm because it can reveal the full joint posterior distribution of the calibrated parameters. However, to date, Bayesian calibration has not been used often in health decision sciences due to practical and computational burdens. In this paper we propose to use artificial neural networks (ANN) as one solution to these limitations. Methods: Bayesian Calibration using Artificial Neural Networks (BayCANN) involves (1) training an ANN metamodel on a sample of model inputs and outputs, and (2) then calibrating th…
The evolution and release of fission gas impacts the performance of UO2 nuclear fuel. We have created a Bayesian framework to calibrate a novel model for fission gas transport that predicts diffusion rates of uranium and xenon in UO2 under both thermal equilibrium and irradiation conditions. Data sets are taken from historical diffusion, gas release, and thermodynamic experiments. These data sets consist invariably of summary statistics, including a measurement value with an associated uncertainty. Our calibration strategy uses synthetic data sets in order to estimate the parameters in the mod…
Item response theory (IRT) is the statistical paradigm underlying a dominant family of generative probabilistic models for test responses, used to quantify traits in individuals relative to target populations. The graded response model (GRM) is a particular IRT model that is used for ordered polytomous test responses. Both the development and the application of the GRM and other IRT models require statistical decisions. For formulating these models (calibration), one needs to decide on methodologies for item selection, inference, and regularization. For applying these models (test scoring), on…
In this article, we present a recently released R package for Bayesian calibration. Many industrial fields are facing unfeasible or costly field experiments. These experiments are replaced with numerical/computer experiments which are realized by running a numerical code. Bayesian calibration intends to estimate, through a posterior distribution, input parameters of the code in order to make the code outputs close to the available experimental data. The code can be time consuming while the Bayesian calibration implies a lot of code calls which makes studies too burdensome. A discrepancy might …
Bayesian approaches for handling covariate measurement error are well established, and yet arguably are still relatively little used by researchers. For some this is likely due to unfamiliarity or disagreement with the Bayesian inferential paradigm. For others a contributory factor is the inability of standard statistical packages to perform such Bayesian analyses. In this paper we first give an overview of the Bayesian approach to handling covariate measurement error, and contrast it with regression calibration (RC), arguably the most commonly adopted approach. We then argue why the Bayesian …
Modeling the relationship between vehicle speed and density on the road is a fundamental problem in traffic flow theory. Recent research found that using the least-squares (LS) method to calibrate single-regime speed-density models is biased because of the uneven distribution of samples. This paper explains the issue of the LS method from a statistical perspective: the biased calibration is caused by the correlations/dependencies in regression residuals. Based on this explanation, we propose a new calibration method for single-regime speed-density models by modeling the covariance of residuals…
The Mars Curiosity rover carries an instrument, ChemCam, designed to measure the composition of surface rocks and soil using laser-induced breakdown spectroscopy (LIBS). The measured spectra from this instrument must be analyzed to identify the component elements in the target sample, as well as their relative proportions. This process, which we call disaggregation, is complicated by so-called matrix effects, which describe nonlinear changes in the relative heights of emission lines as an unknown function of composition due to atomic interactions within the LIBS plasma. In this work we explore…
Acoustic emission (AE) is a widely used technology to study source mechanisms and material properties during high-pressure rock failure experiments. It is important to understand the physical quantities that acoustic emission sensors measure, as well as the response of these sensors as a function of frequency. This study calibrates the newly built AE system in the MIT Rock Physics Laboratory using a ball-bouncing system. Full waveforms of multi-bounce events due to ball drops are used to infer the transfer function of lead zirconate titanate (PZT) sensors in high pressure environments. Uncerta…
This paper addresses the Bayesian calibration of dynamic models with parametric and structural uncertainties, in particular where the uncertain parameters are unknown/poorly known spatio-temporally varying subsystem models. Independent stationary Gaussian processes with uncertain hyper-parameters describe uncertainties of the model structure and parameters while Karhunnen-Loeve expansion is adopted to spectrally represent these Gaussian processes. The Karhunnen-Loeve expansion of a prior Gaussian process is projected on a generalized Polynomial Chaos basis, whereas intrusive Galerkin projectio…
Large scale study of primary school student performance relative to their LMS activity and socioeconomic demographics using a Bayesian Additive Regression Trees containing random effects
In this paper we develop a likelihood-free approach for population calibration, which involves finding distributions of model parameters when fed through the model produces a set of outputs that matches available population data. Unlike most other approaches to population calibration, our method produces uncertainty quantification on the estimated distribution. Furthermore, the method can be applied to any population calibration problem, regardless of whether the model of interest is deterministic or stochastic, or whether the population data is observed with or without measurement error. We d…
