Parallel surrogate optimization algorithms have proven to be efficient methods for solving expensive noisy optimization problems. In this work we develop a new parallel surrogate optimization algorithm (ProSRS), using a novel tree-based "zoom strategy" to improve the efficiency of the algorithm. We prove that if ProSRS is run for sufficiently long, with probability converging to one there will be at least one point among all the evaluations that will be arbitrarily close to the global minimum. We compare our algorithm to several state-of-the-art Bayesian optimization algorithms on a suite of standard benchmark functions and two real machine learning hyperparameter-tuning problems. We find that our algorithm not only achieves significantly faster optimization convergence, but is also 1-4 orders of magnitude cheaper in computational cost.
Global optimization problems whose objective function is expensive to evaluate can be solved effectively by recursively fitting a surrogate function to function samples and minimizing an acquisition function to generate new samples. The acquisition step trades off between seeking for a new optimization vector where the surrogate is minimum (exploitation of the surrogate) and looking for regions of the feasible space that have not yet been visited and that may potentially contain better values of the objective function (exploration of the feasible space). This paper proposes a new global optimi…
Helsinki Institute for Information Technology HIIT（赫尔辛基信息技术研究所）；Aalto University（阿尔托大学）；University of Edinburgh（爱丁堡大学）
We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly forward simulations. We frame the inference task as a sequential Bayesian experimental design problem, where the log-likelihood function is modelled with a hierarchical Gaussian process (GP) surrogate model, which is used to efficiently select additional log-likelihood evaluation…
University of California at Santa Barbara（加州大学圣巴巴拉分校）
We develop adaptive replicated designs for Gaussian process metamodels of stochastic experiments. Adaptive batching is a natural extension of sequential design heuristics with the benefit of replication growing as response features are learned, inputs concentrate, and the metamodeling overhead rises. Motivated by the problem of learning the level set of the mean simulator response we develop four novel schemes: Multi-Level Batching (MLB), Ratchet Batching (RB), Adaptive Batched Stepwise Uncertainty Reduction (ABSUR), Adaptive Design with Stepwise Allocation (ADSA) and Deterministic Design with…
IMT School for Advanced Studies Lucca（卢卡高等研究院）；Dalle Molle Institute for Artificial Intelligence - USI/SUPSI（达勒·莫勒人工智能研究所）
This paper proposes a method for solving optimization problems in which the decision-maker cannot evaluate the objective function, but rather can only express a preference such as "this is better than that" between two candidate decision vectors. The algorithm described in this paper aims at reaching the global optimizer by iteratively proposing the decision maker a new comparison to make, based on actively learning a surrogate of the latent (unknown and perhaps unquantifiable) objective function from past sampled decision vectors and pairwise preferences. The surrogate is fit by means of radi…
This paper analyzes DONE, an online optimization algorithm that iteratively minimizes an unknown function based on costly and noisy measurements. The algorithm maintains a surrogate of the unknown function in the form of a random Fourier expansion (RFE). The surrogate is updated whenever a new measurement is available, and then used to determine the next measurement point. The algorithm is comparable to Bayesian optimization algorithms, but its computational complexity per iteration does not depend on the number of measurements. We derive several theoretical results that provide insight on how…
Applications of ML-Based Surrogates in Bayesian Approaches to Inverse Problems
Izmir University of Economics（伊兹密尔经济大学）；Carnegie Melon University（卡内基梅隆大学）；University of Warwick（华威大学）；Case Western Reserve University（凯斯西储大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Advanced Micro Devices, Inc.（超威半导体公司）
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
We present an adaptive approach to the construction of Gaussian process surrogates for Bayesian inference with expensive-to-evaluate forward models. Our method relies on the fully Bayesian approach to training Gaussian process models and utilizes the expected improvement idea from Bayesian global optimization. We adaptively construct training designs by maximizing the expected improvement in fit of the Gaussian process model to the noisy observational data. Numerical experiments on model problems with synthetic data demonstrate the effectiveness of the obtained adaptive designs compared to the…
Department of Basic Neuroscience, University of Geneva（日内瓦大学基础神经科学系）
Variational Bayesian Monte Carlo (VBMC) is a recently introduced framework that uses Gaussian process surrogates to perform approximate Bayesian inference in models with black-box, non-cheap likelihoods. In this work, we extend VBMC to deal with noisy log-likelihood evaluations, such as those arising from simulation-based models. We introduce new `global' acquisition functions, such as expected information gain (EIG) and variational interquantile range (VIQR), which are robust to noise and can be efficiently evaluated within the VBMC setting. In a novel, challenging, noisy-inference benchmark …
In complex large-scale systems such as climate, important effects are caused by a combination of confounding processes that are not fully observable. The identification of sources from observations of system state is vital for attribution and prediction, which inform critical policy decisions. The difficulty of these types of inverse problems lies in the inability to isolate sources and the cost of simulating computational models. Surrogate models may enable the many-query algorithms required for source identification, but data challenges arise from high dimensionality of the state and source,…
Bayesian optimization is an approach to optimizing objective functions that take a long time (minutes or hours) to evaluate. It is best-suited for optimization over continuous domains of less than 20 dimensions, and tolerates stochastic noise in function evaluations. It builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this surrogate to decide where to sample. In this tutorial, we describe how Bayesian optimization works, including Gau…
Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparameters of various machine learning algorithms. Those methods adopt probabilistic surrogate models like Gaussian processes to approximate and minimize the validation error function of hyperparameter values. However, probabilistic surrogates require accurate estimates of sufficient statistics (e.g., covariance) of the error distribution and thus need many function evaluations with a …
SMT 2.0: A Surrogate Modeling Toolbox with a focus on Hierarchical and Mixed Variables Gaussian Processes
ONERA（法国航空航天研究院）；Université de Toulouse（图卢兹大学）；ISAE-SUPAERO（高等航空航天学院）；Polytechnique Montréal（蒙特利尔理工学院）；German Aerospace Center (DLR)（德国航空航天中心）；University of California San Diego（加州大学圣地亚哥分校）；ICA（高等航空航天学院）；INSA（国立应用科学学院）；CNRS（法国国家科学研究中心）；MINES ALBI（阿尔比矿业学校）；UPS（图卢兹第三大学）；University of Michigan（密歇根大学）
When a black-box optimization objective can only be evaluated with costly or noisy measurements, most standard optimization algorithms are unsuited to find the optimal solution. Specialized algorithms that deal with exactly this situation make use of surrogate models. These models are usually continuous and smooth, which is beneficial for continuous optimization problems, but not necessarily for combinatorial problems. However, by choosing the basis functions of the surrogate model in a certain way, we show that it can be guaranteed that the optimal solution of the surrogate model is integer. …
Datasets in engineering applications are often limited and contaminated, mainly due to unavoidable measurement noise and signal distortion. Thus, using conventional data-driven approaches to build a reliable discriminative model, and further applying this identified surrogate to uncertainty analysis remains to be very challenging. A deep learning approach is presented to provide predictions based on limited and noisy data. To address noise perturbation, the Bayesian learning method that naturally facilitates an automatic updating mechanism is considered to quantify and propagate model uncertai…
A Simple Approximate Bayesian Inference Neural Surrogate for Stochastic Petri Net Models
School of Computing and Augmented Intelligence, Arizona State University（计算与增强智能学院，亚利桑那州立大学）；School of Mathematical and Statistical Sciences, Arizona State University（数学与统计科学学院，亚利桑那州立大学）；School of Life Sciences, Arizona State University（生命科学学院，亚利桑那州立大学）
National Cheng Kung University（国立成功大学）；Wells Fargo（富国银行）；Georgia Institute of Technology（佐治亚理工学院）
Global optimization of expensive functions has important applications in physical and computer experiments. It is a challenging problem to develop efficient optimization scheme, because each function evaluation can be costly and the derivative information of the function is often not available. We propose a novel global optimization framework using adaptive Radial Basis Functions (RBF) based surrogate model via uncertainty quantification. The framework consists of two iteration steps. It first employs an RBF-based Bayesian surrogate model to approximate the true function, where the parameters …
The Gaussian Process (GP)-based surrogate model has the inherent capability of capturing the anomaly arising from limited data, lack of data, missing data, and data inconsistencies (noisy/erroneous data) present in the modeling and simulation component of the digital twin framework, specifically for the accident tolerant fuel (ATF) concepts. However, GP will not be very accurate when we have limited high-fidelity (experimental) data. In addition, it is challenging to apply higher dimensional functions (>20-dimensional function) to approximate predictions with the GP. Furthermore, noisy data or…
We study a relaxed version of the column-sampling problem for the Nyström approximation of kernel matrices, where approximations are defined from multisets of landmark points in the ambient space; such multisets are referred to as Nyström samples. We consider an unweighted variation of the radial squared-kernel discrepancy (SKD) criterion as a surrogate for the classical criteria used to assess the Nyström approximation accuracy; in this setting, we discuss how Nyström samples can be efficiently optimised through stochastic gradient descent. We perform numerical experiments which demonstrate t…
Heteroscedastic Gaussian process regression, based on the concept of chained Gaussian processes, is used to build surrogates to predict site-specific loads on an offshore wind turbine. Stochasticity in the inflow turbulence and irregular waves results in load responses that are best represented as random variables rather than deterministic values. Moreover, the effect of these stochastic sources on the loads depends strongly on the mean environmental conditions -- for instance, at low mean wind speeds, inflow turbulence produces much less variability in loads than at high wind speeds. Statisti…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Probabilistic Bisection Algorithm performs root finding based on knowledge acquired from noisy oracle responses. We consider the generalized PBA setting (G-PBA) where the statistical distribution of the oracle is unknown and location-dependent, so that model inference and Bayesian knowledge updating must be performed simultaneously. To this end, we propose to leverage the spatial structure of a typical oracle by constructing a statistical surrogate for the underlying logistic regression step. We investigate several non-parametric surrogates, including Binomial Gaussian Processes (B-GP), Polyno…
Delta smelt is an endangered fish species in the San Francisco estuary that have shown an overall population decline over the past 30 years. Researchers have developed a stochastic, agent-based simulator to virtualize the system, with the goal of understanding the relative contribution of natural and anthropogenic factors suggested as playing a role in their decline. However, the input configuration space is high-dimensional, running the simulator is time-consuming, and its noisy outputs change nonlinearly in both mean and variance. Getting enough runs to effectively learn input--output dynami…
Rochester Institute of Technology（罗切斯特理工学院）；Dalhousie University（达尔豪斯大学）
Estimation of patient-specific model parameters is important for personalized modeling, although sparse and noisy clinical data can introduce significant uncertainty in the estimated parameter values. This importance source of uncertainty, if left unquantified, will lead to unknown variability in model outputs that hinder their reliable adoptions. Probabilistic estimation model parameters, however, remains an unresolved challenge because standard Markov Chain Monte Carlo sampling requires repeated model simulations that are computationally infeasible. A common solution is to replace the simula…
Improving Clinical Dataset Condensation with Mode Connectivity-based Trajectory Surrogates
Institute of Biomedical Engineering, University of Oxford, UK（生物医学工程研究所，牛津大学，英国）；GlaxoSmithKline, London, UK（葛兰素史克，伦敦，英国）；Nuffield Department of Primary Care Health Sciences, University of Oxford, UK（初级保健健康科学尼弗尔部门，牛津大学，英国）；Oxford-Suzhou Institute of Advanced Research (OSCAR), Suzhou, China（牛津-苏州先进研究所（OSCAR），苏州，中国）
Gaussian process surrogate with physical law-corrected prior for multi-coupled PDEs defined on irregular geometry
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Mathematics and Statistics, Central South University（中南大学数学与统计学院）；Institute of Applied Physics and Computational Mathematics（北京应用物理与计算数学研究所）；Institute of Mathematics, Henan Academy of Sciences（河南省科学院数学研究所）
Driven by increased complexity of dynamical systems, the solution of system of differential equations through numerical simulation in optimization problems has become computationally expensive. This paper provides a smart data driven mechanism to construct low dimensional surrogate models. These surrogate models reduce the computational time for solution of the complex optimization problems by using training instances derived from the evaluations of the true objective functions. The surrogate models are constructed using combination of proper orthogonal decomposition and radial basis functions…
We present a flexible trust region descend algorithm for unconstrained and convexly constrained multiobjective optimization problems. It is targeted at heterogeneous and expensive problems, i.e., problems that have at least one objective function that is computationally expensive. The method is derivative-free in the sense that neither need derivative information be available for the expensive objectives nor are gradients approximated using repeated function evaluations as is the case in finite-difference methods. Instead, a multiobjective trust region approach is used that works similarly to …
A multi-fidelity (MF) active learning method is presented for design optimization problems characterized by noisy evaluations of the performance metrics. Namely, a generalized MF surrogate model is used for design-space exploration, exploiting an arbitrary number of hierarchical fidelity levels, i.e., performance evaluations coming from different models, solvers, or discretizations, characterized by different accuracy. The method is intended to accurately predict the design performance while reducing the computational effort required by simulation-driven design (SDD) to achieve the global opti…
The energy transition entails a rapid uptake of renewable energy sources. Besides physical changes within the grid infrastructure, energy storage devices and their smart operation are key measures to master the resulting challenges like, e.g., a highly fluctuating power generation. For the latter, optimization based control has demonstrated its potential on a microgrid level. However, if a network of coupled microgrids is considered, iterative optimization schemes including several communication rounds are typically used. Here, we propose to replace the optimization on the microgrid level by u…
Virginia Tech（弗吉尼亚理工大学）；Oden Institute for Computational Engineering and Sciences, University of Texas at Austin（德克萨斯大学奥斯汀分校奥登计算工程与科学研究所）
Data assimilation is the process of fusing information from imperfect computer simulations with noisy, sparse measurements of reality to obtain improved estimates of the state or parameters of a dynamical system of interest. The data assimilation procedures used in many geoscience applications, such as numerical weather forecasting, are variants of the our-dimensional variational (4D-Var) algorithm. The cost of solving the underlying 4D-Var optimization problem is dominated by the cost of repeated forward and adjoint model runs. This motivates substituting the evaluations of the physical model…
Standard evolutionary optimization algorithms assume that the evaluation of the objective and constraint functions is straightforward and computationally cheap. However, in many real-world optimization problems, these evaluations involve computationally expensive numerical simulations or physical experiments. Surrogate-assisted evolutionary algorithms (SAEAs) have recently gained increased attention for their performance in solving these types of problems. The main idea of SAEAs is the integration of an evolutionary algorithm with a selected surrogate model that approximates the computationall…
Optimizing expensive black-box systems with limited data is an extremely challenging problem. As a resolution, we present a new surrogate optimization approach by addressing two gaps in prior research -- unimportant input variables and inefficient treatment of uncertainty associated with the black-box output. We first design a new flexible non-interpolating parsimonious surrogate model using a partitioning-based multivariate adaptive regression splines approach, Tree Knot MARS (TK-MARS). The proposed model is specifically designed for optimization by capturing the structure of the function, be…
Hybrid variational quantum algorithms, which combine a classical optimizer with evaluations on a quantum chip, are the most promising candidates to show quantum advantage on current noisy, intermediate-scale quantum (NISQ) devices. The classical optimizer is required to perform well in the presence of noise in the objective function evaluations, or else it becomes the weakest link in the algorithm. We introduce the use of Gaussian Processes (GP) as surrogate models to reduce the impact of noise and to provide high quality seeds to escape local minima, whether real or noise-induced. We build th…
Some popular functions used to test global optimization algorithms have multiple local optima, all with the same value, making them all global optima. It is easy to make them more challenging by fortifying them via adding a localized bump at the location of one of the optima. In previous work the authors illustrated this for the Branin-Hoo function and the popular differential evolution algorithm, showing that the fortified Branin-Hoo required an order of magnitude more function evaluations. This paper examines the effect of fortifying the Branin-Hoo function on surrogate-based optimization, w…
