Deep Gaussian processes (DGPs) are increasingly popular as predictive models in machine learning (ML) for their non-stationary flexibility and ability to cope with abrupt regime changes in training data. Here we explore DGPs as surrogates for computer simulation experiments whose response surfaces exhibit similar characteristics. In particular, we transport a DGP's automatic warping of the input space and full uncertainty quantification (UQ), via a novel elliptical slice sampling (ESS) Bayesian posterior inferential scheme, through to active learning (AL) strategies that distribute runs non-uniformly in the input space -- something an ordinary (stationary) GP could not do. Building up the design sequentially in this way allows smaller training sets, limiting both expensive evaluation of the simulator code and mitigating cubic costs of DGP inference. When training data sizes are kept small through careful acquisition, and with parsimonious layout of latent layers, the framework can be both effective and computationally tractable. Our methods are illustrated on simulation data and two real computer experiments of varying input dimensionality. We provide an open source implementation in the "deepgp" package on CRAN.
Helsinki Institute for Information Technology HIIT（赫尔辛基信息技术研究所）；Aalto University（阿尔托大学）；University of Edinburgh（爱丁堡大学）
We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly forward simulations. We frame the inference task as a sequential Bayesian experimental design problem, where the log-likelihood function is modelled with a hierarchical Gaussian process (GP) surrogate model, which is used to efficiently select additional log-likelihood evaluation…
Department of Biostatistics, University of Oslo（奥斯陆大学生物统计系）；Helsinki Institute for Information Technology HIIT, Department of Computer Science, Aalto University（赫尔辛基信息技术研究所，阿尔托大学计算机科学系）
The computational efficiency of approximate Bayesian computation (ABC) has been improved by using surrogate models such as Gaussian processes (GP). In one such promising framework the discrepancy between the simulated and observed data is modelled with a GP which is further used to form a model-based estimator for the intractable posterior. In this article we improve this approach in several ways. We develop batch-sequential Bayesian experimental design strategies to parallellise the expensive simulations. In earlier work only sequential strategies have been used. Current surrogate-based ABC m…
Markov chain Monte Carlo methods for exponential family models with intractable normalizing constant, such as the exchange algorithm, require simulations of the sufficient statistics at every iteration of the Markov chain, which often result in expensive computations. Surrogate models for the likelihood function have been developed to accelerate inference algorithms in this context. However, these surrogate models tend to be relatively inflexible, and often provide a poor approximation to the true likelihood function. In this article, we propose the use of a warped, gradient-enhanced, Gaussian…
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
We present an adaptive approach to the construction of Gaussian process surrogates for Bayesian inference with expensive-to-evaluate forward models. Our method relies on the fully Bayesian approach to training Gaussian process models and utilizes the expected improvement idea from Bayesian global optimization. We adaptively construct training designs by maximizing the expected improvement in fit of the Gaussian process model to the noisy observational data. Numerical experiments on model problems with synthetic data demonstrate the effectiveness of the obtained adaptive designs compared to the…
Generative Bayesian Computation as a Scalable Alternative to Gaussian Process Surrogates
Booth School of Business, University of Chicago（芝加哥大学商学院）；Department of Systems Engineering and Operations Research, George Mason University（乔治·马歇尔大学系统工程与运营管理系）
Informatics Institute（信息学研究所）；University of Amsterdam（阿姆斯特丹大学）
Scientists often express their understanding of the world through a computationally demanding simulation program. Analyzing the posterior distribution of the parameters given observations (the inverse problem) can be extremely challenging. The Approximate Bayesian Computation (ABC) framework is the standard statistical tool to handle these likelihood free problems, but they require a very large number of simulations. In this work we develop two new ABC sampling algorithms that significantly reduce the number of simulations necessary for posterior inference. Both algorithms use confidence estim…
Active Learning of Piecewise Gaussian Process Surrogates
Department of Industrial and Systems Engineering, University of Washington（华盛顿大学工业与系统工程系）；Materials and Manufacturing Directorate, Air Force Research Laboratory（空军研究实验室材料与制造 Directorate）；Department of Industrial Engineering, Pusan National University（釜山国立大学工业工程系）；Department of Statistics, Virginia Tech（弗吉尼亚理工学院统计系）
Engineers widely use Gaussian process regression framework to construct surrogate models aimed to replace computationally expensive physical models while exploring design space. Thanks to Gaussian process properties we can use both samples generated by a high fidelity function (an expensive and accurate representation of a physical phenomenon) and a low fidelity function (a cheap and coarse approximation of the same physical phenomenon) while constructing a surrogate model. However, if samples sizes are more than few thousands of points, computational costs of the Gaussian process regression b…
We investigate a machine learning approach to option Greeks approximation based on Gaussian process (GP) surrogates. The method takes in noisily observed option prices, fits a nonparametric input-output map and then analytically differentiates the latter to obtain the various price sensitivities. Our motivation is to compute Greeks in cases where direct computation is expensive, such as in local volatility models, or can only ever be done approximately. We provide a detailed analysis of numerous aspects of GP surrogates, including choice of kernel family, simulation design, choice of trend fun…
Johns Hopkins University（约翰斯·霍普金斯大学）；Idaho National Laboratory（爱达荷国家实验室）
Estimating the probability of failure for complex real-world systems using high-fidelity computational models is often prohibitively expensive, especially when the probability is small. Exploiting low-fidelity models can make this process more feasible, but merging information from multiple low-fidelity and high-fidelity models poses several challenges. This paper presents a robust multi-fidelity surrogate modeling strategy in which the multi-fidelity surrogate is assembled using an active learning strategy using an on-the-fly model adequacy assessment set within a subset simulation framework …
Heteroscedastic Gaussian process regression, based on the concept of chained Gaussian processes, is used to build surrogates to predict site-specific loads on an offshore wind turbine. Stochasticity in the inflow turbulence and irregular waves results in load responses that are best represented as random variables rather than deterministic values. Moreover, the effect of these stochastic sources on the loads depends strongly on the mean environmental conditions -- for instance, at low mean wind speeds, inflow turbulence produces much less variability in loads than at high wind speeds. Statisti…
Active learning methods have recently surged in the literature due to their ability to solve complex structural reliability problems within an affordable computational cost. These methods are designed by adaptively building an inexpensive surrogate of the original limit-state function. Examples of such surrogates include Gaussian process models which have been adopted in many contributions, the most popular ones being the efficient global reliability analysis (EGRA) and the active Kriging Monte Carlo simulation (AK-MCS), two milestone contributions in the field. In this paper, we first conduct…
University of California at Santa Barbara（加州大学圣巴巴拉分校）
We develop adaptive replicated designs for Gaussian process metamodels of stochastic experiments. Adaptive batching is a natural extension of sequential design heuristics with the benefit of replication growing as response features are learned, inputs concentrate, and the metamodeling overhead rises. Motivated by the problem of learning the level set of the mean simulator response we develop four novel schemes: Multi-Level Batching (MLB), Ratchet Batching (RB), Adaptive Batched Stepwise Uncertainty Reduction (ABSUR), Adaptive Design with Stepwise Allocation (ADSA) and Deterministic Design with…
Multi-fidelity Gaussian process surrogate modeling for regression problems in physics
School of Computation, Information and Technology, Technische Universität München（慕尼黑工业大学计算、信息与技术学院）；Institute of System Dynamics and Control, German Aerospace Center (DLR)（德国航空航天中心系统动力学与控制研究所）
SMT 2.0: A Surrogate Modeling Toolbox with a focus on Hierarchical and Mixed Variables Gaussian Processes
ONERA（法国航空航天研究院）；Université de Toulouse（图卢兹大学）；ISAE-SUPAERO（高等航空航天学院）；Polytechnique Montréal（蒙特利尔理工学院）；German Aerospace Center (DLR)（德国航空航天中心）；University of California San Diego（加州大学圣地亚哥分校）；ICA（高等航空航天学院）；INSA（国立应用科学学院）；CNRS（法国国家科学研究中心）；MINES ALBI（阿尔比矿业学校）；UPS（图卢兹第三大学）；University of Michigan（密歇根大学）
We consider the estimation of sensitivity indices based on divergence measures such as Hellinger distance. For sensitivity analysis of complex models, these divergence-based indices can be estimated by Monte-Carlo sampling (MCS) in combination with kernel density estimation (KDE). In a direct approach, the complex model must be evaluated at every input point generated by MCS, resulting in samples in the input-output space that can be used for density estimation. However, if the computational cost of the complex model strongly limits the number of model evaluations, this direct method gives lar…
Conformal Approach To Gaussian Process Surrogate Evaluation With Coverage Guarantees
EDF R&D（法国电力集团研发部）；Paris-Saclay University（巴黎-萨克雷大学）；CNRS（法国国家科学研究中心）；Laboratoire Interdisciplinaire des Sciences du Numérique（数字科学跨学科实验室）；ENS Paris-Saclay（巴黎-萨克雷高等师范学校）；Centre Borelli（博雷利中心）；Quantmetry（Quantmetry公司）；INRIA（法国国家信息与自动化研究所）；ENSIIE（法国信息与能源工程高等学院）；SINCLAIR AI Lab.（SINCLAIR人工智能实验室）；Institut de Mathématiques de Toulouse（图卢兹数学研究所）
A common challenge in computer experiments and related fields is to efficiently explore the input space using a small number of samples, i.e., the experimental design problem. Much of the recent focus in the computer experiment literature, where modeling is often via Gaussian process (GP) surrogates, has been on space-filling designs, via maximin distance, Latin hypercube, etc. However, it is easy to demonstrate empirically that such designs disappoint when the model hyperparameterization is unknown, and must be estimated from data observed at the chosen design sites. This is true even when th…
Simulation models are widely used in practice to facilitate decision-making in a complex, dynamic and stochastic environment. But they are computationally expensive to execute and optimize, due to lack of analytical tractability. Simulation optimization is concerned with developing efficient sampling schemes -- subject to a computational budget -- to solve such optimization problems. To mitigate the computational burden, surrogates are often constructed using simulation outputs to approximate the response surface of the simulation model. In this tutorial, we provide an up-to-date overview of s…
The quantification of uncertainties of computer simulations due to input parameter uncertainties is paramount to assess a model's credibility. For computationally expensive simulations, this is often feasible only via surrogate models that are learned from a small set of simulation samples. The surrogate models are commonly chosen and deemed trustworthy based on heuristic measures, and substituted for the simulation in order to approximately propagate the simulation input uncertainties to the simulation output. In the process, the contribution of the uncertainties of the surrogate itself to th…
By mixing the target posterior distribution with a surrogate distribution, of which the normalizing constant is tractable, we propose a method for estimating the marginal likelihood using the Wang-Landau algorithm. We show that a faster convergence of the proposed method can be achieved via the momentum acceleration. Two implementation strategies are detailed: (i) facilitating global jumps between the posterior and surrogate distributions via the Multiple-try Metropolis; (ii) constructing the surrogate via the variational approximation. When a surrogate is difficult to come by, we describe a n…
Several applications such as nuclear forensics, nuclear fuel cycle simulations and sensitivity analysis require methods to quickly compute spent fuel nuclide compositions for various irradiation histories. Traditionally, this has been done by interpolating between one-group cross-sections that have been pre-computed from nuclear reactor simulations for a grid of input parameters, using fits such as Cubic Spline. We propose the use of Gaussian Processes (GP) to create surrogate models, which not only provide nuclide compositions, but also the gradient and estimates of their prediction uncertain…
Multifidelity-Augmented Gaussian Process Inputs for Surrogate Modeling from Scarce Data
School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA（航空航天工程学院，佐治亚理工学院，亚特兰大，GA）；School of Computational Science & Engineering, Georgia Institute of Technology, Atlanta, GA（计算科学与工程学院，佐治亚理工学院，亚特兰大，GA）；Air Force Research Lab, Wright-Patterson AFB, Dayton, Ohio（空军研究实验室，威格伍德-帕特森空军基地，代顿，俄亥俄）
Surrogate models have shown to be an extremely efficient aid in solving engineering problems that require repeated evaluations of an expensive computational model. They are built by sparsely evaluating the costly original model and have provided a way to solve otherwise intractable problems. A crucial aspect in surrogate modelling is the assumption of smoothness and regularity of the model to approximate. This assumption is however not always met in reality. For instance in civil or mechanical engineering, some models may present discontinuities or non-smoothness, e.g., in case of instability …
IMT School for Advanced Studies Lucca（卢卡高等研究院）；Dalle Molle Institute for Artificial Intelligence - USI/SUPSI（达勒·莫勒人工智能研究所）
This paper proposes a method for solving optimization problems in which the decision-maker cannot evaluate the objective function, but rather can only express a preference such as "this is better than that" between two candidate decision vectors. The algorithm described in this paper aims at reaching the global optimizer by iteratively proposing the decision maker a new comparison to make, based on actively learning a surrogate of the latent (unknown and perhaps unquantifiable) objective function from past sampled decision vectors and pairwise preferences. The surrogate is fit by means of radi…
Gaussian process surrogate with physical law-corrected prior for multi-coupled PDEs defined on irregular geometry
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Mathematics and Statistics, Central South University（中南大学数学与统计学院）；Institute of Applied Physics and Computational Mathematics（北京应用物理与计算数学研究所）；Institute of Mathematics, Henan Academy of Sciences（河南省科学院数学研究所）
Helsinki Institute for Information Technology（赫尔辛基信息技术研究所）；Aalto University（阿尔托大学）；University of Oslo（奥斯陆大学）；University of Helsinki（赫尔辛基大学）；University of Manchester（曼彻斯特大学）
In recent years, surrogate models have been successfully used in likelihood-free inference to decrease the number of simulator evaluations. The current state-of-the-art performance for this task has been achieved by Bayesian Optimization with Gaussian Processes (GPs). While this combination works well for unimodal target distributions, it is restricting the flexibility and applicability of Bayesian Optimization for accelerating likelihood-free inference more generally. We address this problem by proposing a Deep Gaussian Process (DGP) surrogate model that can handle more irregularly behaved ta…
A problem of considerable importance within the field of uncertainty quantification (UQ) is the development of efficient methods for the construction of accurate surrogate models. Such efforts are particularly important to applications constrained by high-dimensional uncertain parameter spaces. The difficulty of accurate surrogate modeling in such systems, is further compounded by data scarcity brought about by the large cost of forward model evaluations. Traditional response surface techniques, such as Gaussian process regression (or Kriging) and polynomial chaos are difficult to scale to hig…
Data assimilation is widely used to improve flood forecasting capability, especially through parameter inference requiring statistical information on the uncertain input parameters (upstream discharge, friction coefficient) as well as on the variability of the water level and its sensitivity with respect to the inputs. For particle filter or ensemble Kalman filter, stochastically estimating probability density function and covariance matrices from a Monte Carlo random sampling requires a large ensemble of model evaluations, limiting their use in real-time application. To tackle this issue, fas…
This paper presents a new Gaussian process (GP) surrogate modeling for predicting the outcome of a physical experiment where some experimental inputs are controlled by other manipulating factors. Particularly, we are interested in the case where the control precision is not very high, so the input factor values vary significantly even under the same setting of the corresponding manipulating factors. The case is observed in our main application to carbon nanotube growth experiments, where one experimental input among many is manipulated by another manipulating factors, and the relation between …
American University of Beirut（贝鲁特美国大学）；New York University（纽约大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
A new algorithm is developed to tackle the issue of sampling non-Gaussian model parameter posterior probability distributions that arise from solutions to Bayesian inverse problems. The algorithm aims to mitigate some of the hurdles faced by traditional Markov Chain Monte Carlo (MCMC) samplers, through constructing proposal probability densities that are both, easy to sample and that provide a better approximation to the target density than a simple Gaussian proposal distribution would. To achieve that, a Gaussian proposal distribution is augmented with a Gaussian Process (GP) surface that hel…
The state-of-the-art linked Gaussian process offers a way to build analytical emulators for systems of computer models. We generalize the closed form expressions for the linked Gaussian process under the squared exponential kernel to a class of Matérn kernels, that are essential in advanced applications. An iterative procedure to construct linked Gaussian processes as surrogate models for any feed-forward systems of computer models is presented and illustrated on a feed-back coupled satellite system. We also introduce an adaptive design algorithm that could increase the approximation accuracy …
Bayesian posterior distributions arising in modern applications, including inverse problems in partial differential equation models in tomography and subsurface flow, are often computationally intractable due to the large computational cost of evaluating the data likelihood. To alleviate this problem, we consider using Gaussian process regression to build a surrogate model for the likelihood, resulting in an approximate posterior distribution that is amenable to computations in practice. This work serves as an introduction to Gaussian process regression, in particular in the context of buildin…
A Streaming Sparse Cholesky Method for Derivative-Informed Gaussian Process Surrogates Within Digital Twin Applications
organization= Department of Mechanical Engineering, University of Utah , addressline= , city= Salt Lake City , postcode= 84112 , state= UT , country= USA；organization= Kahlert School of Computing, University of Utah , city= Salt Lake City , postcode= 84112 , state= UT , country= USA；organization= Scientific Computing \& Imaging Institute, University of Utah , addressline= , city= Salt Lake City , postcode= 84112 , state= UT , country= USA
