Response probability distribution estimation of expensive computer simulators: A Bayesian active learning perspective using Gaussian process regression
Chao Dang, Marcos A. Valdebenito, Nataly A. Manque, Jun Xu, Matthias G. R. Faes
Estimation of the response probability distributions of computer simulators in the presence of randomness is a crucial task in many fields. However, achieving this task with guaranteed accuracy remains an open computational challenge, especially for expensive-to-evaluate computer simulators. In this work, a Bayesian active learning perspective is presented to address the challenge, which is based on the use of the Gaussian process (GP) regression. First, estimation of the response probability distributions is conceptually interpreted as a Bayesian inference problem, as opposed to frequentist inference. This interpretation provides several important benefits: (1) it quantifies and propagates discretization error probabilistically; (2) it incorporates prior knowledge of the computer simulator, and (3) it enables the effective reduction of numerical uncertainty in the solution to a prescribed level. The conceptual Bayesian idea is then realized by using the GP regression, where we derive the posterior statistics of the response probability distributions in semi-analytical form and also provide a numerical solution scheme. Based on the practical Bayesian approach, a Bayesian active learning (BAL) method is further proposed for estimating the response probability distributions. In this context, the key contribution lies in the development of two crucial components for active learning, i.e., stopping criterion and learning function, by taking advantage of posterior statistics. It is empirically demonstrated by five numerical examples that the proposed BAL method can efficiently estimate the response probability distributions with desired accuracy.
The scientific understanding of real-world processes has dramatically improved over the years through computer simulations. Such simulators represent complex mathematical models that are implemented as computer codes which are often expensive. The validity of using a particular simulator to draw accurate conclusions relies on the assumption that the computer code is correctly calibrated. This calibration procedure is often pursued under extensive experimentation and comparison with data from a real-world process. The problem is that the data collection may be so expensive that only a handful o…
An emulator is a fast-to-evaluate statistical approximation of a detailed mathematical model (simulator). When used in lieu of simulators, emulators can expedite tasks that require many repeated evaluations, such as sensitivity analyses, policy optimization, model calibration, and value-of-information analyses. Emulators are developed using the output of simulators at specific input values (design points). Developing an emulator that closely approximates the simulator can require many design points, which becomes computationally expensive. We describe a self-terminating active learning algorit…
Multifidelity Simulation-based Inference for Computationally Expensive Simulators
Department of Computer Science, KU Leuven（KU莱文大学计算机科学系）；VIB Center for AI and Computational Biology (VIB.AI)（VIB人工智能与计算生物学中心）；VIB-KU Leuven Center for Neuroscience（VIB-KU莱文神经科学中心）；Gatsby Computational Neuroscience Unit, UCL（Gatsby计算神经科学单元，UCL）；Machine Learning in Science, University of Tübingen（科学机器学习，图宾根大学）；Tübingen AI Center（图宾根人工智能中心）；Max Planck Institute for Biological Intelligence, Martinsried（生物智能马克斯·普朗克研究所，马尔茨里德）；Department of Electrical Engineering, KU Leuven（KU莱文电子工程系）
We provide a method to solve optimization problem when objective function is a complex stochastic simulator of an urban transportation system. To reach this goal, a Bayesian optimization framework is introduced. We show how the choice of prior and inference algorithm effect the outcome of our optimization procedure. We develop dimensionality reduction techniques that allow for our optimization techniques to be applicable for real-life problems. We develop a distributed, Gaussian Process Bayesian regression and active learning models that allow parallel execution of our algorithms and enable us…
Tensor-variate Gaussian process regression for efficient emulation of complex systems: comparing regressor and covariance structures in outer product and parallel partial emulators
Cavendish Laboratory, University of Cambridge（剑桥大学卡文迪许实验室）；University of Cambridge（剑桥大学）；Cavendish Laboratory（卡文迪许实验室）
The estimation of unknown values of parameters (or hidden variables, control variables) that characterise a physical system often relies on the comparison of measured data with synthetic data produced by some numerical simulator of the system as the parameter values are varied. This process often encounters two major difficulties: the generation of synthetic data for each considered set of parameter values can be computationally expensive if the system model is complicated; and the exploration of the parameter space can be inefficient and/or incomplete, a typical example being when the explora…
Efficient optimization of expensive black-box simulators via marginal means, with application to neutrino detector design
Department of Statistical Science, Duke University（杜克大学统计科学系）；Nuclear Science Division, Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室核科学部）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
This article focuses on the multi-objective optimization of stochastic simulators with high output variance, where the input space is finite and the objective functions are expensive to evaluate. We rely on Bayesian optimization algorithms, which use probabilistic models to make predictions about the functions to be optimized. The proposed approach is an extension of the Pareto Active Learning (PAL) algorithm for the estimation of Pareto-optimal solutions that makes it suitable for the stochastic setting. We named it Pareto Active Learning for Stochastic Simulators (PALS). The performance of P…
Computer models, also known as simulators, can be computationally expensive to run, and for this reason statistical surrogates, known as emulators, are often used. Any statistical model, including an emulator, should be validated before being used, otherwise resulting decisions can be misguided. We discuss how current methods for validating Gaussian process emulators of deterministic models are insufficient for emulators of stochastic computer models and develop a framework for diagnosing problems in stochastic emulators. These diagnostics are based on independently validating the mean and var…
Bayesian optimization (BO) and its batch extensions are successful for optimizing expensive black-box functions. However, these traditional BO approaches are not yet ideal for optimizing less expensive functions when the computational cost of BO can dominate the cost of evaluating the blackbox function. Examples of these less expensive functions are cheap machine learning models, inexpensive physical experiment through simulators, and acquisition function optimization in Bayesian optimization. In this paper, we consider a batch BO setting for situations where function evaluations are less expe…
We present mlrMBO, a flexible and comprehensive R toolbox for model-based optimization (MBO), also known as Bayesian optimization, which addresses the problem of expensive black-box optimization by approximating the given objective function through a surrogate regression model. It is designed for both single- and multi-objective optimization with mixed continuous, categorical and conditional parameters. Additional features include multi-point batch proposal, parallelization, visualization, logging and error-handling. mlrMBO is implemented in a modular fashion, such that single components can b…
Accelerating Posterior Inference from Pulsar Light Curves via Learned Latent Representations and Local Simulator-Guided Optimization
Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Florida State University（佛罗里达州立大学）；New Mexico State University（新墨西哥州立大学）；NASA Goddard Space Flight Center（美国宇航局戈达德太空飞行中心）
Large amounts of labeled data are typically required to train deep learning models. For many real-world problems, however, acquiring additional data can be expensive or even impossible. We present semi-supervised deep kernel learning (SSDKL), a semi-supervised regression model based on minimizing predictive variance in the posterior regularization framework. SSDKL combines the hierarchical representation learning of neural networks with the probabilistic modeling capabilities of Gaussian processes. By leveraging unlabeled data, we show improvements on a diverse set of real-world regression tas…
The case-cohort study design bypasses resource constraints by collecting certain expensive covariates for only a small subset of the full cohort. Weighted Cox regression is the most widely used approach for analysing case-cohort data within the Cox model, but is inefficient. Alternative approaches based on multiple imputation and nonparametric maximum likelihood suffer from incompatibility and computational issues respectively. We introduce a novel Bayesian framework for case-cohort Cox regression that avoids the aforementioned problems. Users can include auxiliary variables to help predict th…
Oregon State University（俄勒冈州立大学）；Oak Ridge National Laboratory（橡树岭国家实验室）
We propose a nested weighted Tchebycheff Multi-objective Bayesian optimization framework where we build a regression model selection procedure from an ensemble of models, towards better estimation of the uncertain parameters of the weighted-Tchebycheff expensive black-box multi-objective function. In existing work, a weighted Tchebycheff MOBO approach has been demonstrated which attempts to estimate the unknown utopia in formulating acquisition function, through calibration using a priori selected regression model. However, the existing MOBO model lacks flexibility in selecting the appropriate…
Collected data, which is used for analysis or prediction tasks, often have a hierarchical structure, for example, data from various people performing the same task. Modeling the data's structure can improve the reliability of the derived results and prediction performance of newly unobserved data. Bayesian modeling provides a tool-kit for designing hierarchical models. However, Markov Chain Monte Carlo methods which are commonly used for parameter estimation are computationally expensive. This often renders its use for many applications not applicable. However, variational Bayesian methods all…
University of Toronto（多伦多大学）；Harvard University（哈佛大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；Atinary Technologies Inc（Atinary科技公司）；Canadian Institute for Advanced Research (CIFAR)（加拿大高级研究院）；Vector Institute（矢量研究所）
Bayesian optimization has emerged as a powerful strategy to accelerate scientific discovery by means of autonomous experimentation. However, expensive measurements are required to accurately estimate materials properties, and can quickly become a hindrance to exhaustive materials discovery campaigns. Here, we introduce Gemini: a data-driven model capable of using inexpensive measurements as proxies for expensive measurements by correcting systematic biases between property evaluation methods. We recommend using Gemini for regression tasks with sparse data and in an autonomous workflow setting …
University of Oxford（牛津大学）；Improbable（Improbable公司）；Durham University（杜伦大学）
Simulation models of complex dynamics in the natural and social sciences commonly lack a tractable likelihood function, rendering traditional likelihood-based statistical inference impossible. Recent advances in machine learning have introduced novel algorithms for estimating otherwise intractable likelihood functions using a likelihood ratio trick based on binary classifiers. Consequently, efficient likelihood approximations can be obtained whenever good probabilistic classifiers can be constructed. We propose a kernel classifier for sequential data using path signatures based on the recently…
Norwegian University of Science and Technology（挪威科技大学）；Delft University of Technology（代尔夫特理工大学）
This article investigates the optimization of yaw control inputs of a nine-turbine wind farm. The wind farm is simulated using the high-fidelity simulator SOWFA. The optimization is performed with a modifier adaptation scheme based on Gaussian processes. Modifier adaptation corrects for the mismatch between plant and model and helps to converge to the actual plan optimum. In the case study the modifier adaptation approach is compared with the Bayesian optimization approach. Moreover, the use of two different covariance functions in the Gaussian process regression is discussed. Practical recomm…
Oilfield development related decisions are made using reservoir simulation-based optimization study in which different production scenarios and well controls are compared. Such simulations are computationally expensive and so surrogate models are used to accelerate studies. Deep learning has been used in past to generate surrogates, but such models often fail to quantify prediction uncertainty and are not interpretable. In this work, beta-VAE based regression is proposed to generate simulation surrogates for use in optimization workflow. beta-VAE enables interpretable, factorized representatio…
Computer simulators are often used as a substitute of complex real-life phenomena which are either expensive or infeasible to experiment with. This paper focuses on how to efficiently solve the inverse problem for an expensive to evaluate time series valued computer simulator. The research is motivated by a hydrological simulator which has to be tuned for generating realistic rainfall-runoff measurements in Athens, Georgia, USA. Assuming that the simulator returns g(x,t) over L time points for a given input x, the proposed methodology begins with a careful construction of a discretization (tim…
Gaussian process regression is a popular Bayesian framework for surrogate modeling of expensive data sources. As part of a broader effort in scientific machine learning, many recent works have incorporated physical constraints or other a priori information within Gaussian process regression to supplement limited data and regularize the behavior of the model. We provide an overview and survey of several classes of Gaussian process constraints, including positivity or bound constraints, monotonicity and convexity constraints, differential equation constraints provided by linear PDEs, and boundar…
A key objective in engineering problems is to predict an unknown experimental surface over an input domain. In complex physical experiments, this may be hampered by response censoring, which results in a significant loss of information. For such problems, experimental design is paramount for maximizing predictive power using a small number of expensive experimental runs. To tackle this, we propose a novel adaptive design method, called the integrated censored mean-squared error (ICMSE) method. The ICMSE method first estimates the posterior probability of a new observation being censored, then …
Computer simulators are nowadays widely used to understand complex physical systems in many areas such as aerospace, renewable energy, climate modeling, and manufacturing. One fundamental issue in the study of computer simulators is known as experimental design, that is, how to select the input settings where the computer simulator is run and the corresponding response is collected. Extra care should be taken in the selection process because computer simulators can be computationally expensive to run. The selection shall acknowledge and achieve the goal of the analysis. This article focuses on…
We here consider the subset simulation method which approaches a failure event using a decreasing sequence of nested intermediate failure events. The method resembles importance sampling, which actively explores a probability space by conditioning the next evaluation on the previous evaluations using a Markov chain Monte Carlo (MCMC) algorithm. A Markov chain typically requires many steps to estimate the target distribution, which is impractical with expensive numerical models. Therefore, we propose to approximate each step of a Markov chain locally with Gaussian process (GP) regression. Bench…
MF-GLaM: A multifidelity stochastic emulator using generalized lambda models
Chair of Risk, Safety and Uncertainty Quantification, ETH Zürich（苏黎世联邦理工学院风险、安全与不确定性量化教席）；Laboratory of Signals and Systems, CentraleSupélec, Université Paris-Saclay（巴黎萨克雷大学中央理工-高等电力学院信号与系统实验室）
