We address the problem of training models with black-box and hard-to-optimize metrics by expressing the metric as a monotonic function of a small number of easy-to-optimize surrogates. We pose the training problem as an optimization over a relaxed surrogate space, which we solve by estimating local gradients for the metric and performing inexact convex projections. We analyze gradient estimates based on finite differences and local linear interpolations, and show convergence of our approach under smoothness assumptions with respect to the surrogates. Experimental results on classification and ranking problems verify the proposal performs on par with methods that know the mathematical formulation, and adds notable value when the form of the metric is unknown.
This paper investigates the control of an ML component within the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) devoted to black-box optimization. The known CMA-ES weakness is its sample complexity, the number of evaluations of the objective function needed to approximate the global optimum. This weakness is commonly addressed through surrogate optimization, learning an estimate of the objective function a.k.a. surrogate model, and replacing most evaluations of the true objective function with the (inexpensive) evaluation of the surrogate model. This paper presents a principled cont…
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；AXA（安盛）；Polish Academy of Science（波兰科学院）
Local surrogate models, to approximate the local decision boundary of a black-box classifier, constitute one approach to generate explanations for the rationale behind an individual prediction made by the back-box. This paper highlights the importance of defining the right locality, the neighborhood on which a local surrogate is trained, in order to approximate accurately the local black-box decision boundary. Unfortunately, as shown in this paper, this issue is not only a parameter or sampling distribution challenge and has a major impact on the relevance and quality of the approximation of t…
Learning Surrogates for Offline Black-Box Optimization via Gradient Matching
Princeton University（普林斯顿大学）；Lewis-Sigler Institute of Integrative Genomics, Princeton University（普林斯顿大学刘易斯-西格勒整合基因组学研究所）；Washington State University（华盛顿州立大学）；School of Electrical Engineering and Computer Science, Washington State University（华盛顿州立大学电气与计算机工程学院）
Czech Technical University（捷克技术大学）；Charles University in Prague（布拉格查理大学）；Czech Academy of Sciences（捷克科学院）
Surrogate modeling has become a valuable technique for black-box optimization tasks with expensive evaluation of the objective function. In this paper, we investigate the relationship between the predictive accuracy of surrogate models and features of the black-box function landscape. We also study properties of features for landscape analysis in the context of different transformations and ways of selecting the input data. We perform the landscape analysis of a large set of data generated using runs of a surrogate-assisted version of the Covariance Matrix Adaptation Evolution Strategy on the …
A challenging problem in both engineering and computer science is that of minimising a function for which we have no mathematical formulation available, that is expensive to evaluate, and that contains continuous and integer variables, for example in automatic algorithm configuration. Surrogate-based algorithms are very suitable for this type of problem, but most existing techniques are designed with only continuous or only discrete variables in mind. Mixed-Variable ReLU-based Surrogate Modelling (MVRSM) is a surrogate-based algorithm that uses a linear combination of rectified linear units, d…
Imperial College London（帝国理工学院）；SLAC National Accelerator Laboratory（SLAC国家加速器实验室）；University of Oxford（牛津大学）
We propose a novel method for gradient-based optimization of black-box simulators using differentiable local surrogate models. In fields such as physics and engineering, many processes are modeled with non-differentiable simulators with intractable likelihoods. Optimization of these forward models is particularly challenging, especially when the simulator is stochastic. To address such cases, we introduce the use of deep generative models to iteratively approximate the simulator in local neighborhoods of the parameter space. We demonstrate that these local surrogates can be used to approximate…
MF-CLIP: Leveraging CLIP as Surrogate Models for No-box Adversarial Attacks
Hong Kong University of Science and Technology（香港科技大学）；University of Naples Federico II（那不勒斯费德里科二世大学）；Singapore Management University（新加坡管理大学）；Beijing Jiaotong University（北京交通大学）；National Lab of Pattern Recognition（模式识别国家重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
KU Leuven（鲁汶大学）；University of Amsterdam（阿姆斯特丹大学）；Université Laval（拉瓦尔大学）
Highly regulated industries, like banking and insurance, ask for transparent decision-making algorithms. At the same time, competitive markets are pushing for the use of complex black box models. We therefore present a procedure to develop a Model-Agnostic Interpretable Data-driven suRRogate (maidrr) suited for structured tabular data. Knowledge is extracted from a black box via partial dependence effects. These are used to perform smart feature engineering by grouping variable values. This results in a segmentation of the feature space with automatic variable selection. A transparent generali…
AXA（安盛集团）；OECD（经济合作与发展组织）；Sorbonne Universite（索邦大学）；CNRS（法国国家科学研究中心）；Polish Academy of Science（波兰科学院）
Interpretable surrogates of black-box predictors trained on high-dimensional tabular datasets can struggle to generate comprehensible explanations in the presence of correlated variables. We propose a model-agnostic interpretable surrogate that provides global and local explanations of black-box classifiers to address this issue. We introduce the idea of concepts as intuitive groupings of variables that are either defined by a domain expert or automatically discovered using correlation coefficients. Concepts are embedded in a surrogate decision tree to enhance its comprehensibility. First expe…
Harvard University（哈佛大学）；Radcliffe Institute for Advanced Study, Harvard University（哈佛大学拉德克利夫高等研究院）；Indian Institute of Science（印度科学院）
The area under the ROC curve (AUC) is a widely used performance measure in machine learning. Increasingly, however, in several applications, ranging from ranking to biometric screening to medicine, performance is measured not in terms of the full area under the ROC curve, but in terms of the \emph{partial} area under the ROC curve between two false positive rates. In this paper, we develop support vector algorithms for directly optimizing the partial AUC between any two false positive rates. Our methods are based on minimizing a suitable proxy or surrogate objective for the partial AUC error. …
Receiver Operating Characteristic (ROC) curves are plots of true positive rate versus false positive rate which are useful for evaluating binary classification models, but difficult to use for learning since the Area Under the Curve (AUC) is non-convex. ROC curves can also be used in other problems that have false positive and true positive rates such as changepoint detection. We show that in this more general context, the ROC curve can have loops, points with highly sub-optimal error rates, and AUC greater than one. This observation motivates a new optimization objective: rather than maximizi…
Warsaw University of Technology（华沙理工大学）；University of Warsaw（华沙大学）
Complex black-box predictive models may have high accuracy, but opacity causes problems like lack of trust, lack of stability, sensitivity to concept drift. On the other hand, interpretable models require more work related to feature engineering, which is very time consuming. Can we train interpretable and accurate models, without timeless feature engineering? In this article, we show a method that uses elastic black-boxes as surrogate models to create a simpler, less opaque, yet still accurate and interpretable glass-box models. New models are created on newly engineered features extracted/le…
A recent line of work on black-box adversarial attacks has revived the use of transfer from surrogate models by integrating it into query-based search. However, we find that existing approaches of this type underperform their potential, and can be overly complicated besides. Here, we provide a short and simple algorithm which achieves state-of-the-art results through a search which uses the surrogate network's class-score gradients, with no need for other priors or heuristics. The guiding assumption of the algorithm is that the studied networks are in a fundamental sense learning similar funct…
Area under ROC (AUC) is an important metric for binary classification and bipartite ranking problems. However, it is difficult to directly optimizing AUC as a learning objective, so most existing algorithms are based on optimizing a surrogate loss to AUC. One significant drawback of these surrogate losses is that they require pairwise comparisons among training data, which leads to slow running time and increasing local storage for online learning. In this work, we describe a new surrogate loss based on a reformulation of the AUC risk, which does not require pairwise comparison but rankings of…
Surrogate explainers of black-box machine learning predictions are of paramount importance in the field of eXplainable Artificial Intelligence since they can be applied to any type of data (images, text and tabular), are model-agnostic and are post-hoc (i.e., can be retrofitted). The Local Interpretable Model-agnostic Explanations (LIME) algorithm is often mistakenly unified with a more general framework of surrogate explainers, which may lead to a belief that it is the solution to surrogate explainability. In this paper we empower the community to "build LIME yourself" (bLIMEy) by proposing a…
When a black-box optimization objective can only be evaluated with costly or noisy measurements, most standard optimization algorithms are unsuited to find the optimal solution. Specialized algorithms that deal with exactly this situation make use of surrogate models. These models are usually continuous and smooth, which is beneficial for continuous optimization problems, but not necessarily for combinatorial problems. However, by choosing the basis functions of the surrogate model in a certain way, we show that it can be guaranteed that the optimal solution of the surrogate model is integer. …
In this paper, we propose an offline counterfactual policy estimation framework called Genie to optimize Sponsored Search Marketplace. Genie employs an open box simulation engine with click calibration model to compute the KPI impact of any modification to the system. From the experimental results on Bing traffic, we showed that Genie performs better than existing observational approaches that employs randomized experiments for traffic slices that have frequent policy updates. We also show that Genie can be used to tune completely new policies efficiently without creating risky randomized expe…
Machine learning classifiers are critically prone to evasion attacks. Adversarial examples are slightly modified inputs that are then misclassified, while remaining perceptively close to their originals. Last couple of years have witnessed a striking decrease in the amount of queries a black box attack submits to the target classifier, in order to forge adversarials. This particularly concerns the black-box score-based setup, where the attacker has access to top predicted probabilites: the amount of queries went from to millions of to less than a thousand. This paper presents SurFree, a geomet…
JD Technology（京东科技）；University of California, Los Angeles（加州大学洛杉矶分校）
We consider adversarial attacks to a black-box model when no queries are allowed. In this setting, many methods directly attack surrogate models and transfer the obtained adversarial examples to fool the target model. Plenty of previous works investigated what kind of attacks to the surrogate model can generate more transferable adversarial examples, but their performances are still limited due to the mismatches between surrogate models and the target model. In this paper, we tackle this problem from a novel angle -- instead of using the original surrogate models, can we obtain a Meta-Surrogat…
Shanghai Jiao Tong University（上海交通大学）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）
This paper addresses the challenging black-box adversarial attack problem, where only classification confidence of a victim model is available. Inspired by consistency of visual saliency between different vision models, a surrogate model is expected to improve the attack performance via transferability. By combining transferability-based and query-based black-box attack, we propose a surprisingly simple baseline approach (named SimBA++) using the surrogate model, which significantly outperforms several state-of-the-art methods. Moreover, to efficiently utilize the query feedback, we update the…
Black-box and preference-based optimization algorithms are global optimization procedures that aim to find the global solutions of an optimization problem using, respectively, the least amount of function evaluations or sample comparisons as possible. In the black-box case, the analytical expression of the objective function is unknown and it can only be evaluated through a (costly) computer simulation or an experiment. In the preference-based case, the objective function is still unknown but it corresponds to the subjective criterion of an individual. So, it is not possible to quantify such c…
In this paper, a new sequential surrogate-based optimization (SSBO) algorithm is developed, which aims to improve the global search ability and local search efficiency for the global optimization of expensive black-box models. The proposed method involves three basic sub-criteria to infill new samples asynchronously to balance the global exploration and local exploitation. First, to capture the promising possible global optimal region, searching for the global optimum with genetic algorithm (GA) based on the current surrogate models of the objective and constraint functions. Second, to infill …
Efficient Black-box Adversarial Attacks via Bayesian Optimization Guided by a Function Prior
Tsinghua University（清华大学）；JQ Investments（JQ投资）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；RealAI；Kaiyuan International Mathematical Sciences Institute（开源国际数学科学研究院）
Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Bayesian optimization (BO) has demonstrated potential for optimizing control performance in data-limited settings, especially for systems with unknown dynamics or unmodeled performance objectives. The BO algorithm efficiently trades-off exploration and exploitation by leveraging uncertainty estimates using surrogate models. These surrogates are usually learned using data collected from the target dynamical system to be optimized. Intuitively, the convergence rate of BO is better for surrogate models that can accurately predict the target system performance. In classical BO, initial surrogate m…
EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functions
School of Industrial Engineering, Eindhoven University of Technology（埃因霍温理工大学工业工程学院）；Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology（代尔夫特理工大学电气工程、数学与计算机科学学院）
Kangwon National University（江原大学）；Bionsight, Inc.（Bionsight公司）
The $F_β$ score is a commonly used measure of classification performance, which plays crucial roles in classification tasks with imbalanced data sets. However, the $F_β$ score cannot be used as a loss function by gradient-based learning algorithms for optimizing neural network parameters due to its non-differentiability. On the other hand, commonly used loss functions such as the binary cross-entropy (BCE) loss are not directly related to performance measures such as the $F_β$ score, so that neural networks optimized by using the loss functions may not yield optimal performance measures. In th…
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；University of California, Berkeley（加州大学伯克利分校）
Building surrogate models is one common approach when we attempt to learn unknown black-box functions. Bayesian optimization provides a framework which allows us to build surrogate models based on sequential samples drawn from the function and find the optimum. Tuning algorithmic parameters to optimize the performance of large, complicated "black-box" application codes is a specific important application, which aims at finding the optima of black-box functions. Within the Bayesian optimization framework, the Gaussian process model produces smooth or continuous sample paths. However, the black-…
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Tencent AI Lab（腾讯人工智能实验室）
This work studies black-box adversarial attacks against deep neural networks (DNNs), where the attacker can only access the query feedback returned by the attacked DNN model, while other information such as model parameters or the training datasets are unknown. One promising approach to improve attack performance is utilizing the adversarial transferability between some white-box surrogate models and the target model (i.e., the attacked model). However, due to the possible differences on model architectures and training datasets between surrogate and target models, dubbed "surrogate biases", t…
Microsoft Research India（微软研究院印度）；Indian Institute of Science（印度科学学院）
The problem of maximizing precision at the top of a ranked list, often dubbed Precision@k (prec@k), finds relevance in myriad learning applications such as ranking, multi-label classification, and learning with severe label imbalance. However, despite its popularity, there exist significant gaps in our understanding of this problem and its associated performance measure. The most notable of these is the lack of a convex upper bounding surrogate for prec@k. We also lack scalable perceptron and stochastic gradient descent algorithms for optimizing this performance measure. In this paper we make …
Surrogate modeling for interpreting black-box LLMs in medical predictions
Medical Big Data Research Center, Seoul National University Medical Research Center, Seoul National University College of Medicine（首尔国立大学医学院医学大数据研究中心，首尔国立大学医学院）；Department of Biomedical Systems Informatics, Yonsei University College of Medicine（延世大学医学院生物医学系统信息学系）；Laboratory for Computational Physiology, Massachusetts Institute of Technology（麻省理工学院计算生理学实验室）；Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center（贝斯以色列德aconess医疗中心呼吸科、重症医学科和睡眠医学科）；Department of Biostatistics, Harvard T.H. Chan School of Public Health（哈佛大学T.H. Chan公共卫生学院生物统计学系）；Department of Cardiology, Yongin Severance Hospital, Yonsei University College of Medicine（延世大学医学院永宁松甫医院心内科）；Center for Digital Health, Yongin Severance Hospital, Yonsei University Health System（延世大学健康系统永宁松甫医院数字健康中心）；Institute for Innovation in Digital Healthcare, Severance Hospital, Seoul, Republic of Korea（首尔松甫医院数字医疗创新研究所）
ST Engineering Electronics-SUTD Cyber Security Laboratory（新科工程电子-新加坡科技设计大学网络安全实验室）；Singapore University of Technology and Design（新加坡科技设计大学）；Ben-Gurion University of the Negev（本-古里安大学）；Deutsche Telekom Innovation Laboratories at Ben-Gurion University of the Negev（本-古里安大学德国电信创新实验室）
We investigate to what extent alternative variants of Artificial Neural Networks (ANNs) are susceptible to adversarial attacks. We analyse the adversarial robustness of conventional, stochastic ANNs and Spiking Neural Networks (SNNs) in the raw image space, across three different datasets. Our experiments reveal that stochastic ANN variants are almost equally as susceptible as conventional ANNs when faced with simple iterative gradient-based attacks in the white-box setting. However we observe, that in black-box settings, stochastic ANNs are more robust than conventional ANNs, when faced with …
PMBO: Enhancing Black-Box Optimization through Multivariate Polynomial Surrogates
CASUS – Center for Advanced Systems Understanding（CASUS——高级系统理解中心）；Helmholtz-Zentrum Dresden-Rossendorf e.V. (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；California Institute of Technology（加州理工学院）；University of Wrocław（弗罗茨瓦夫大学）
UIUC（伊利诺伊大学厄巴纳-香槟分校）；Google Research USA（谷歌美国研究院）；Google Research Accra（谷歌阿克拉研究院）
We consider learning to optimize a classification metric defined by a black-box function of the confusion matrix. Such black-box learning settings are ubiquitous, for example, when the learner only has query access to the metric of interest, or in noisy-label and domain adaptation applications where the learner must evaluate the metric via performance evaluation using a small validation sample. Our approach is to adaptively learn example weights on the training dataset such that the resulting weighted objective best approximates the metric on the validation sample. We show how to model and est…
CPU simulators are useful tools for modeling CPU execution behavior. However, they suffer from inaccuracies due to the cost and complexity of setting their fine-grained parameters, such as the latencies of individual instructions. This complexity arises from the expertise required to design benchmarks and measurement frameworks that can precisely measure the values of parameters at such fine granularity. In some cases, these parameters do not necessarily have a physical realization and are therefore fundamentally approximate, or even unmeasurable. In this paper we present DiffTune, a system fo…
Nanjing University（南京大学）；the University of Iowa（爱荷华大学）
NDCG, namely Normalized Discounted Cumulative Gain, is a widely used ranking metric in information retrieval and machine learning. However, efficient and provable stochastic methods for maximizing NDCG are still lacking, especially for deep models. In this paper, we propose a principled approach to optimize NDCG and its top-$K$ variant. First, we formulate a novel compositional optimization problem for optimizing the NDCG surrogate, and a novel bilevel compositional optimization problem for optimizing the top-$K$ NDCG surrogate. Then, we develop efficient stochastic algorithms with provable co…
University of Bristol（布里斯托大学）；AXA（安盛）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Polish Academy of Science（波兰科学院）
Local surrogate approaches for explaining machine learning model predictions have appealing properties, such as being model-agnostic and flexible in their modelling. Several methods exist that fit this description and share this goal. However, despite their shared overall procedure, they set out different objectives, extract different information from the black-box, and consequently produce diverse explanations, that are -- in general -- incomparable. In this work we review the similarities and differences amongst multiple methods, with a particular focus on what information they extract from …
University of Bristol（布里斯托大学）；AXA（安盛）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Polish Academy of Science（波兰科学院）
This paper analyses the fundamental ingredients behind surrogate explanations to provide a better understanding of their inner workings. We start our exposition by considering global surrogates, describing the trade-off between complexity of the surrogate and fidelity to the black-box being modelled. We show that transitioning from global to local - reducing coverage - allows for more favourable conditions on the Pareto frontier of fidelity-complexity of a surrogate. We discuss the interplay between complexity, fidelity and coverage, and consider how different user needs can lead to problem fo…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables. Our method uses gradients of a neural network trained jointly with model parameters or policies, and is applicable in both discrete and continuous settings. We demonstrate this framework for training discr…
