Any supervised machine learning analysis is required to provide an estimate of the out-of-sample predictive performance. However, it is imperative to also provide a quantification of the uncertainty of this performance in the form of a confidence or credible interval (CI) and not just a point estimate. In an AutoML setting, estimating the CI is challenging due to the ``winner's curse", i.e., the bias of estimation due to cross-validating several machine learning pipelines and selecting the winning one. In this work, we perform a comparative evaluation of 9 state-of-the-art methods and variants in CI estimation in an AutoML setting on a corpus of real and simulated datasets. The methods are compared in terms of inclusion percentage (does a 95\% CI include the true performance at least 95\% of the time), CI tightness (tighter CIs are preferable as being more informative), and execution time. The evaluation is the first one that covers most, if not all, such methods and extends previous work to imbalanced and small-sample tasks. In addition, we present a variant, called BBC-F, of an existing method (the Bootstrap Bias Correction, or BBC) that maintains the statistical properties of the BBC but is more computationally efficient. The results support that BBC-F and BBC dominate the other methods in all metrics measured.
In practice, machine learning (ML) workflows require various different steps, from data preprocessing, missing value imputation, model selection, to model tuning as well as model evaluation. Many of these steps rely on human ML experts. AutoML - the field of automating these ML pipelines - tries to help practitioners to apply ML off-the-shelf without any expert knowledge. Most modern AutoML systems like auto-sklearn, H20-AutoML or TPOT aim for high predictive performance, thereby generating ensembles that consist almost exclusively of black-box models. This, in turn, makes the interpretation f…
Position: A Call to Action for a Human-Centered AutoML Paradigm
L3S Research Center（L3S研究中心）；Ludwig-Maximilians-Universität München（慕尼黑大学）；Fraunhofer Institute for Integrated Circuits IIS, Fraunhofer IIS（弗劳恩霍夫集成电路研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Microsoft（微软公司）；Albert-Ludwigs-Universität Freiburg（阿尔伯特-路德维希-弗莱堡大学）
AutoML for Multi-Class Anomaly Compensation of Sensor Drift
Institute for Information Processing (tnt), Leibniz University Hannover（汉诺威莱布尼茨大学信息处理研究所（tnt））；Department of Electrical and Computer Engineering, University of Southern California（南加州大学电气与计算机工程系）；Institute for Artificial Intelligence, Leibniz University Hannover（汉诺威莱布尼茨大学人工智能研究所）
Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in AutoML
University of Siegen（锡根大学）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；Leiden University（莱顿大学）；RWTH Aachen（亚琛工业大学）；University of British Columbia（不列颠哥伦比亚大学）
College of Transportation Engineering, Tongji University（同济大学交通运输工程学院）；University of California, Berkeley（加利福尼亚大学伯克利分校）；Institute for Infocomm Research, Agency for Science Technology and Research (A*STAR)（新加坡科技研究局信息通信研究所）
With the enrichment of smartphones, driving distractions caused by phone usages have become a threat to driving safety. A promising way to mitigate driving distractions is to detect them and give real-time safety warnings. However, existing detection algorithms face two major challenges, low user acceptance caused by in-vehicle camera sensors, and uncertain accuracy of pre-trained models due to drivers individual differences. Therefore, this study proposes a domain-specific automated machine learning (AutoML) to self-learn the optimal models to detect distraction based on lane-keeping performa…
Fujitsu Research of America, Inc.（美国富士通研究有限公司）；Fujitsu Ltd.（富士通有限公司）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Automatic machine learning, or AutoML, holds the promise of truly democratizing the use of machine learning (ML), by substantially automating the work of data scientists. However, the huge combinatorial search space of candidate pipelines means that current AutoML techniques, generate sub-optimal pipelines, or none at all, especially on large, complex datasets. In this work we propose an AutoML technique SapientML, that can learn from a corpus of existing datasets and their human-written pipelines, and efficiently generate a high-quality pipeline for a predictive task on a new dataset. To comb…
Automated Machine Learning for Remaining Useful Life Predictions
USU Software AG（USU软件公司）；Esslingen University of Applied Sciences（埃斯林根应用科学大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）；University of Stuttgart（斯图加特大学）；Fraunhofer IPA（弗劳恩霍夫制造工程与自动化研究所）
Towards Evolutionary-based Automated Machine Learning for Small Molecule Pharmacokinetic Prediction
Baker Heart and Diabetes Institute（贝克心脏与糖尿病研究所）；School of Chemistry & Molecular Biosciences, The University of Queensland（昆士兰大学化学与分子生物科学学院）；The University of Melbourne（墨尔本大学）
Automated machine learning techniques benefited from tremendous research progress in recently. These developments and the continuous-growing demand for machine learning experts led to the development of numerous AutoML tools. However, these tools assume that the entire training dataset is available upfront and that the underlying distribution does not change over time. These assumptions do not hold in a data stream mining setting where an unbounded stream of data cannot be stored and is likely to manifest concept drift. Industry applications of machine learning on streaming data become more po…
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH.（德国人工智能研究中心有限公司）；Institute of Informatics, Ludwig-Maximilians-Universität München（慕尼黑大学信息学院）
This paper presents an experimental comparison among four Automated Machine Learning (AutoML) methods for recommending the best classification algorithm for a given input dataset. Three of these methods are based on Evolutionary Algorithms (EAs), and the other is Auto-WEKA, a well-known AutoML method based on the Combined Algorithm Selection and Hyper-parameter optimisation (CASH) approach. The EA-based methods build classification algorithms from a single machine learning paradigm: either decision-tree induction, rule induction, or Bayesian network classification. Auto-WEKA combines algorithm…
AutoML systems are currently rising in popularity, as they can build powerful models without human oversight. They often combine techniques from many different sub-fields of machine learning in order to find a model or set of models that optimize a user-supplied criterion, such as predictive performance. The ultimate goal of such systems is to reduce the amount of time spent on menial tasks, or tasks that can be solved better by algorithms while leaving decisions that require human intelligence to the end-user. In recent years, the importance of other criteria, such as fairness and interpretab…
Columbia University（哥伦比亚大学）；University of Chicago（芝加哥大学）
We introduce an automatic machine learning (AutoML) modeling architecture called Autostacker, which combines an innovative hierarchical stacking architecture and an Evolutionary Algorithm (EA) to perform efficient parameter search. Neither prior domain knowledge about the data nor feature preprocessing is needed. Using EA, Autostacker quickly evolves candidate pipelines with high predictive accuracy. These pipelines can be used as is or as a starting point for human experts to build on. Autostacker finds innovative combinations and structures of machine learning models, rather than selecting a…
Deep neural networks are behind many of the recent successes in machine learning applications. However, these models can produce overconfident decisions while encountering out-of-distribution (OOD) examples or making a wrong prediction. This inconsistent predictive confidence limits the integration of independently-trained learning models into a larger system. This paper introduces separable concept learning framework to realistically measure the performance of classifiers in presence of OOD examples. In this setup, several instances of a classifier are trained on different parts of a partitio…
Numerous algorithms have been proposed for detecting anomalies (outliers, novelties) in an unsupervised manner. Unfortunately, it is not trivial, in general, to understand why a given sample (record) is labelled as an anomaly and thus diagnose its root causes. We propose the following reduced-dimensionality, surrogate model approach to explain detector decisions: approximate the detection model with another one that employs only a small subset of features. Subsequently, samples can be visualized in this low-dimensionality space for human understanding. To this end, we develop PROTEUS, an AutoM…
Automated Machine Learning in Radiomics: A Comparative Evaluation of Performance, Efficiency and Accessibility
University of Valencia（瓦伦西亚大学）；Research & Frontiers in AI Department, Quantitative Imaging Biomarkers in Medicine, Quibim SL（研究与前沿人工智能部门、定量影像生物标志物在医学中的应用、Quibim SL）；Intelligent Data Analysis Laboratory, IDAL, University of Valencia（智能数据分析实验室，IDAL，瓦伦西亚大学）
General predictive models do not provide a measure of confidence in predictions without Bayesian assumptions. A way to circumvent potential restrictions is to use conformal methods for constructing non-parametric confidence regions, that offer guarantees regarding validity. In this paper we provide a detailed description of a computationally efficient conformal procedure for Kernel Ridge Regression (KRR), and conduct a comparative numerical study to see how well conformal regions perform against the Bayesian confidence sets. The results suggest that conformalized KRR can yield predictive confi…
A Conformal Predictive Measure for Assessing Catastrophic Forgetting
Communication Systems Department, EURECOM, France（EURECOM通信系统部）；Andalusian Institute of Data Science and Computational Intelligence (DaSCI)（安达卢西亚数据科学与计算智能研究所）；Department of Computer Science and Artificial Intelligence, University of Granada, Spain（格拉纳达大学计算机科学与人工智能系）
Trust-Aware Predictive Emissions Monitoring for Gas Turbine Fleets with Limited Labelled Data
School of Natural and Computing Sciences University of Aberdeen（自然与计算科学学院 布拉德福德大学）；School of Computing Sciences University of East Anglia（计算科学学院 东安格利亚大学）；Siemens Energy Industrial Turbomachinery Ltd.（西门子能源工业涡轮机有限公司）；Department of Physics and Technology UiT The Arctic University of Norway（物理与技术系 UiT 北极大学）
Miami University（迈阿密大学）；Air Force Research Lab（空军研究实验室）
Assessing the predictive accuracy of black box classifiers is challenging in the absence of labeled test datasets. In these scenarios we may need to rely on a human oracle to evaluate individual predictions; presenting the challenge to create query algorithms to guide the search for points that provide the most information about the classifier's predictive characteristics. Previous works have focused on developing utility models and query algorithms for discovering unknown unknowns --- misclassifications with a predictive confidence above some arbitrary threshold. However, if misclassification…
University Clinic Tübingen（图宾根大学医院）；University of Tübingen（图宾根大学）
Motor disturbances can affect the interaction with dynamic objects, such as catching a ball. A classification of clinical catching trials might give insight into the existence of pathological alterations in the relation of arm and ball movements. Accurate, but also early decisions are required to classify a catching attempt before the catcher's first ball contact. To obtain clinically valuable results, a significant decision confidence of at least 75% is required. Hence, three competing objectives have to be optimized at the same time: accuracy, earliness and decision-making confidence. Here w…
Visual Geometry Group, University of Oxford（牛津大学视觉几何组）；Visym Labs（Visym实验室）
Knowing when an output can be trusted is critical for reliably using face recognition systems. While there has been enormous effort in recent research on improving face verification performance, understanding when a model's predictions should or should not be trusted has received far less attention. Our goal is to assign a confidence score for a face image that reflects its quality in terms of recognizable information. To this end, we propose a method for generating image quality training data automatically from 'mated-pairs' of face images, and use the generated data to train a lightweight Pr…
High-risk domains require reliable confidence estimates from predictive models. Deep latent variable models provide these, but suffer from the rigid variational distributions used for tractable inference, which err on the side of overconfidence. We propose Stochastic Quantized Activation Distributions (SQUAD), which imposes a flexible yet tractable distribution over discretized latent variables. The proposed method is scalable, self-normalizing and sample efficient. We demonstrate that the model fully utilizes the flexible distribution, learns interesting non-linearities, and provides predicti…
UC Berkeley（加州大学伯克利分校）；Amazon（亚马逊）；Toyota Research Institute（丰田研究所）
Recent work has shown that the performance of machine learning models can vary substantially when models are evaluated on data drawn from a distribution that is close to but different from the training distribution. As a result, predicting model performance on unseen distributions is an important challenge. Our work connects techniques from domain adaptation and predictive uncertainty literature, and allows us to predict model accuracy on challenging unseen distributions without access to labeled data. In the context of distribution shift, distributional distances are often used to adapt model…
Western Sydney University（西悉尼大学）；Ecole Polytechnique（巴黎综合理工学院）；Institut Polytechnique de Paris（巴黎理工学院）
A multi-label classifier estimates the binary label state (relevant vs irrelevant) for each of a set of concept labels, for any given instance. Probabilistic multi-label classifiers provide a predictive posterior distribution over all possible labelset combinations of such label states (the powerset of labels) from which we can provide the best estimate, simply by selecting the labelset corresponding to the largest expected accuracy, over that distribution. For example, in maximizing exact match accuracy, we provide the mode of the distribution. But how does this relate to the confidence we ma…
IT University of Copenhagen（哥本哈根信息技术大学）；Technical University of Denmark（丹麦技术大学）
We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on…
Deep learning models achieve high predictive accuracy across a broad spectrum of tasks, but rigorously quantifying their predictive uncertainty remains challenging. Usable estimates of predictive uncertainty should (1) cover the true prediction targets with high probability, and (2) discriminate between high- and low-confidence prediction instances. Existing methods for uncertainty quantification are based predominantly on Bayesian neural networks; these may fall short of (1) and (2) -- i.e., Bayesian credible intervals do not guarantee frequentist coverage, and approximate posterior inference…
Existing machine learning approaches for data-driven predictive maintenance are usually black boxes that claim high predictive power yet cannot be understood by humans. This limits the ability of humans to use these models to derive insights and understanding of the underlying failure mechanisms, and also limits the degree of confidence that can be placed in such a system to perform well on future data. We consider the task of predicting hard drive failure in a data center using recent algorithms for interpretable machine learning. We demonstrate that these methods provide meaningful insights …
In this work, a machine learning approach is developed for predicting the outcomes of football matches. The novelty of this research lies in the utilisation of the Kelly Index to first classify matches into categories where each one denotes the different levels of predictive difficulty. Classification models using a wide suite of algorithms were developed for each category of matches in order to determine the efficacy of the approach. In conjunction to this, a set of previously unexplored features were engineering including Elo-based variables. The dataset originated from the Premier League ma…
Generalized Conformal Predictive Systems Under Distributional Shifts
IDLab Seminar for Statistics（统计研究所研讨会）；Department of Electronics（电子系）；ETH Zurich（苏黎世联邦理工学院）；Information Systems Zurich, Switzerland（苏黎世信息系统，瑞士）；Ghent University（根特大学）
We develop conformal prediction methods for constructing valid predictive confidence sets in multiclass and multilabel problems without assumptions on the data generating distribution. A challenge here is that typical conformal prediction methods---which give marginal validity (coverage) guarantees---provide uneven coverage, in that they address easy examples at the expense of essentially ignoring difficult examples. By leveraging ideas from quantile regression, we build methods that always guarantee correct coverage but additionally provide (asymptotically optimal) conditional coverage for bo…
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. Our main insight is …
BioMedIA Group, Department of Computing, Imperial College London, UK（伦敦帝国理工学院计算系BioMedIA小组）；School of Biomedical Engineering & Imaging Sciences, King’s College London, UK（伦敦国王学院生物医学工程与成像科学学院）；Guys and St Thomas’ Hospital, London, UK（伦敦盖伊和圣托马斯医院）
Evaluation of predictive deep learning (DL) models beyond conventional performance metrics has become increasingly important for applications in sensitive environments like healthcare. Such models might have the capability to encode and analyse large sets of data but they often lack comprehensive interpretability methods, preventing clinical trust in predictive outcomes. Quantifying uncertainty of a prediction is one way to provide such interpretability and promote trust. However, relatively little attention has been paid to how to include such requirements into the training of the model. In t…
Nagoya University（名古屋大学）；Georgia Institute of Technology（佐治亚理工学院）；RIKEN（理化学研究所）
In predictive modeling for high-stake decision-making, predictors must be not only accurate but also reliable. Conformal prediction (CP) is a promising approach for obtaining the confidence of prediction results with fewer theoretical assumptions. To obtain the confidence set by so-called full-CP, we need to refit the predictor for all possible values of prediction results, which is only possible for simple predictors. For complex predictors such as random forests (RFs) or neural networks (NNs), split-CP is often employed where the data is split into two parts: one part for fitting and another…
