A key question in evaluation of computer models is Does the computer model adequately represent reality? A six-step process for computer model validation is set out in Bayarri et al. [Technometrics 49 (2007) 138--154] (and briefly summarized below), based on comparison of computer model runs with field data of the process being modeled. The methodology is particularly suited to treating the major issues associated with the validation process: quantifying multiple sources of error and uncertainty in computer models; combining multiple sources of information; and being able to adapt to different, but related scenarios. Two complications that frequently arise in practice are the need to deal with highly irregular functional data and the need to acknowledge and incorporate uncertainty in the inputs. We develop methodology to deal with both complications. A key part of the approach utilizes a wavelet representation of the functional data, applies a hierarchical version of the scalar validation methodology to the wavelet coefficients, and transforms back, to ultimately compare computer model output with field output. The generality of the methodology is only limited by the capability of a combination of computational tools and the appropriateness of decompositions of the sort (wavelets) employed here. The methods and analyses we present are illustrated with a test bed dynamic stress analysis for a particular engineering system.
When computer codes are used for modeling complex physical systems, their unknown parameters are tuned by calibration techniques. A discrepancy function may be added to the computer code in order to capture its discrepancy with the real physical process. By considering the validation question of a computer code as a Bayesian selection model problem, Damblin et al. (2016) have highlighted a possible confounding effect in certain configurations between the code discrepancy and a linear computer code by using a Bayesian testing procedure based on the intrinsic Bayes factor. In this paper, we inve…
The analysis of computer models can be aided by the construction of surrogate models, or emulators, that statistically model the numerical computer model. Increasingly, computer models are becoming stochastic, yielding different outputs each time they are run, even if the same input values are used. Stochastic computer models are more difficult to analyse and more difficult to emulate - often requiring substantially more computer model runs to fit. We present a method of using deterministic approximations of the computer model to better construct an emulator. The method is applied to numerous …
Bayesian calibration of computer models tunes unknown input parameters by comparing outputs with observations. For model outputs that are distributed over space, this becomes computationally expensive because of the output size. To overcome this challenge, we employ a basis representation of the model outputs and observations: we match these decompositions to carry out the calibration efficiently. In the second step, we incorporate the non-stationary behaviour, in terms of spatial variations of both variance and correlations, in the calibration. We insert two integrated nested Laplace approxim…
Calibration of expensive computer models with high-dimensional output fields can be approached via history matching. If the entire output field is matched, with patterns or correlations between locations or time points represented, calculating the distance metric between observational data and model output for a single input setting requires a time intensive inversion of a high-dimensional matrix. By using a low-dimensional basis representation rather than emulating each output individually, we define a metric in the reduced space that allows the implausibility for the field to be calculated e…
The calibration of complex computer codes using uncertainty quantification (UQ) methods is a rich area of statistical methodological development. When applying these techniques to simulators with spatial output, it is now standard to use principal component decomposition to reduce the dimensions of the outputs in order to allow Gaussian process emulators to predict the output for calibration. We introduce the `terminal case', in which the model cannot reproduce observations to within model discrepancy, and for which standard calibration methods in UQ fail to give sensible results. We show that…
In many real-world applications we are interested in approximating costly functions that are analytically unknown, e.g. complex computer codes. An emulator provides a fast approximation of such functions relying on a limited number of evaluations. Gaussian processes (GPs) are commonplace emulators due to their statistical properties such as the ability to estimate their own uncertainty. GPs are essentially developed to fit smooth, continuous functions. However, the assumptions of continuity and smoothness is unwarranted in many situations. For example, in computer models where bifurcations or …
Classification is a vital tool that is important for modelling many complex numerical models. A model or system may be such that, for certain areas of input space, the output either does not exist, or is not in a quantifiable form. Here, we present a new method for classification where the model outputs are given distinct classifying labels, which we model using a latent Gaussian process (GP). The latent variable is estimated using MCMC sampling, a unique likelihood and distinct prior specifications. Our classifier is then verified by calculating a misclassification rate across the input space…
Computer experiments can emulate the physical systems, help computational investigations, and yield analytic solutions. They have been widely employed with many engineering applications (e.g., aerospace, automotive, energy systems. Conventional Bayesian optimization did not incorporate the nested structures in computer experiments. This paper proposes a novel nested Bayesian optimization for complex computer experiments with multi-step or hierarchical characteristics. We prove the theoretical properties of nested outputs given two cases: Gaussian or non-Gaussian. The closed forms of nested exp…
The recent accelerated growth in the computing power has generated popularization of experimentation with dynamic computer models in various physical and engineering applications. Despite the extensive statistical research in computer experiments, most of the focus had been on the theoretical and algorithmic innovations for the design and analysis of computer models with scalar responses. In this paper, we propose a computationally efficient statistical emulator for a large-scale dynamic computer simulator (i.e., simulator which gives time series outputs). The main idea is to first find a good…
Hurricane-driven storm surge is one of the most deadly and costly natural disasters, making precise quantification of the surge hazard of great importance. Surge hazard quantification is often performed through physics-based computer models of storm surges. Such computer models can be implemented with a wide range of fidelity levels, with computational burdens varying by several orders of magnitude due to the nature of the system. The threat posed by surge makes greater fidelity highly desirable, however such models and their high-volume output tend to come at great computational cost, which c…
The construction of computer models (mathematical models implemented in computer codes), with respect to observed phenomena, is usually undertaken by building different variants depending on modeller sensibility, and choosing the one yielding the best fit of the field data, according to Root Mean Squared Error (RMSE) based measures. Usually a particular model is chosen because of its marginally lower RMSE, and not because of its actual higher adequacy, risking that its capability of extrapolating predictions is poor. This work aims at improving the current practice in the creation of computer …
Sepsis is a poorly understood and potentially life-threatening complication that can occur as a result of infection. Early detection and treatment improves patient outcomes, and as such it poses an important challenge in medicine. In this work, we develop a flexible classifier that leverages streaming lab results, vitals, and medications to predict sepsis before it occurs. We model patient clinical time series with multi-output Gaussian processes, maintaining uncertainty about the physiological state of a patient while also imputing missing values. The mean function takes into account the effe…
There are many practical difficulties in the calibration of computer models to experimental data. One such complication is the fact that certain combinations of the calibration inputs can cause the code to output data lacking fundamental properties, or even to produce no output at all. In many cases the researchers want or need to exclude the possibility of these "failures" within their analyses. We propose a Bayesian (meta-)model in which the posterior distribution for the calibration parameters naturally excludes regions of the input space corresponding to failed runs. That is, we define a s…
Observing system uncertainty experiments (OSUEs) have been recently proposed as a cost-effective way to perform probabilistic assessment of retrievals for NASA's Orbiting Carbon Observatory-2 (OCO-2) mission. One important component in the OCO-2 retrieval algorithm is a full-physics forward model that describes the mathematical relationship between atmospheric variables such as carbon dioxide and radiances measured by the remote sensing instrument. This forward model is complicated and computationally expensive but large-scale OSUEs require evaluation of this model numerous times, which makes …
