Automated machine learning (AutoML) systems aim at finding the best machine learning (ML) pipeline that automatically matches the task and data at hand. We investigate the robustness of machine learning pipelines generated with three AutoML systems, TPOT, H2O, and AutoKeras. In particular, we study the influence of dirty data on accuracy, and consider how using dirty training data may help create more robust solutions. Furthermore, we also analyze how the structure of the generated pipelines differs in different cases.
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；CISPA - Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts. This led to the development of many sophisticated test procedures going beyond the standard supervised learning frameworks, whose usage can require specialized knowledge about two-sample testing. We use a simple test that takes the mean discrepancy of a witness function as the test statistic and prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML. Without any u…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Several AutoML approaches have been proposed to automate the machine learning (ML) process, such as searching for the ML model architectures and hyper-parameters. However, these AutoML pipelines only focus on improving the learning accuracy of benign samples while ignoring the ML model robustness under adversarial attacks. As ML systems are increasingly being used in a variety of mission-critical applications, improving the robustness of ML systems has become of utmost importance. In this paper, we propose the first robust AutoML framework, Robusta--based on reinforcement learning (RL)--to per…
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
Automatic machine learning (AutoML) is a key enabler of the mass deployment of the next generation of machine learning systems. A key desideratum for future ML systems is the automatic selection of models and hyperparameters. We present a novel method of selecting performant configurations for a given task by performing offline autoML and mining over a diverse set of tasks. By mining the training tasks, we can select a compact portfolio of configurations that perform well over a wide variety of tasks, as well as learn a strategy to select portfolio configurations for yet-unseen tasks. The algo…
A common problem machine learning developers are faced with is overfitting, that is, fitting a pipeline too closely to the training data that the performance degrades for unseen data. Automated machine learning aims to free (or at least ease) the developer from the burden of pipeline creation, but this overfitting problem can persist. In fact, this can become more of a problem as we look to iteratively optimise the performance of an internal cross-validation (most often \textit{k}-fold). While this internal cross-validation hopes to reduce this overfitting, we show we can still risk overfittin…
Developing, scaling, and deploying modern Machine Learning solutions remains challenging for small- and middle-sized enterprises (SMEs). This is due to a high entry barrier of building and maintaining a dedicated IT team as well as the difficulties of real-world data (RWD) compared to standard benchmark data. To address this challenge, we discuss the implementation and concepts of Chameleon, a semi-AutoML framework. The goal of Chameleon is fast and scalable development and deployment of production-ready machine learning systems into the workflow of SMEs. We first discuss the RWD challenges fa…
AutoML for Multi-Class Anomaly Compensation of Sensor Drift
Institute for Information Processing (tnt), Leibniz University Hannover（汉诺威莱布尼茨大学信息处理研究所（tnt））；Department of Electrical and Computer Engineering, University of Southern California（南加州大学电气与计算机工程系）；Institute for Artificial Intelligence, Leibniz University Hannover（汉诺威莱布尼茨大学人工智能研究所）
Automated Machine Learning has grown very successful in automating the time-consuming, iterative tasks of machine learning model development. However, current methods struggle when the data is imbalanced. Since many real-world datasets are naturally imbalanced, and improper handling of this issue can lead to quite useless models, this issue should be handled carefully. This paper first introduces a new benchmark to study how different AutoML methods are affected by label imbalance. Second, we propose strategies to better deal with imbalance and integrate them into an existing AutoML framework.…
Analyzing better time series with limited human effort is of interest to academia and industry. Driven by business scenarios, we organized the first Automated Time Series Regression challenge (AutoSeries) for the WSDM Cup 2020. We present its design, analysis, and post-hoc experiments. The code submission requirement precluded participants from any manual intervention, testing automated machine learning capabilities of solutions, across many datasets, under hardware and time limitations. We prepared 10 datasets from diverse application domains (sales, power consumption, air quality, traffic, a…
Adversarial robustness is one of the essential safety criteria for guaranteeing the reliability of machine learning models. While various adversarial robustness testing approaches were introduced in the last decade, we note that most of them are incompatible with non-differentiable models such as tree ensembles. Since tree ensembles are widely used in industry, this reveals a crucial gap between adversarial robustness research and practical applications. This paper proposes a novel whitebox adversarial robustness testing approach for tree ensemble models. Concretely, the proposed approach smoo…
While the demand for machine learning (ML) applications is booming, there is a scarcity of data scientists capable of building such models. Automatic machine learning (AutoML) approaches have been proposed that help with this problem by synthesizing end-to-end ML data processing pipelines. However, these follow a best-effort approach and a user in the loop is necessary to curate and refine the derived pipelines. Since domain experts often have little or no expertise in machine learning, easy-to-use interactive interfaces that guide them throughout the model building process are necessary. In t…
Beihang University（北京航空航天大学）；Yangzhou Collaborative Innovation Research Institute CO., LTD（扬州协同创新研究院有限公司）；No. 38 Research Institute of CETC（中国电子科技集团公司第三十八研究所）
While deep neural networks (DNNs) have strengthened the performance of cooperative multi-agent reinforcement learning (c-MARL), the agent policy can be easily perturbed by adversarial examples. Considering the safety critical applications of c-MARL, such as traffic management, power management and unmanned aerial vehicle control, it is crucial to test the robustness of c-MARL algorithm before it was deployed in reality. Existing adversarial attacks for MARL could be used for testing, but is limited to one robustness aspects (e.g., reward, state, action), while c-MARL model could be attacked fr…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Technion, Israel Institute of Technology（以色列理工学院）
We derive generalization bounds for learning algorithms based on their robustness: the property that if a testing sample is "similar" to a training sample, then the testing error is close to the training error. This provides a novel approach, different from the complexity or stability arguments, to study generalization of learning algorithms. We further show that a weak notion of robustness is both sufficient and necessary for generalizability, which implies that robustness is a fundamental property for learning algorithms to work.
Recently, various Deep Neural Network (DNN) models have been proposed for environments like embedded systems with stringent energy constraints. The fundamental problem of determining the robustness of a DNN with respect to its energy consumption (energy robustness) is relatively unexplored compared to accuracy-based robustness. This work investigates the energy robustness of Adaptive Neural Networks (AdNNs), a type of energy-saving DNNs proposed for many energy-sensitive domains and have recently gained traction. We propose EREBA, the first black-box testing method for determining the energy r…
ETH Zürich（苏黎世联邦理工学院）；Shanghai Jiao Tong University（上海交通大学）；University of Michigan-Shanghai Jiao Tong University Joint Institute（上海交通大学密歇根学院）；University of Illinois（伊利诺伊大学）
Quantum machine learning models have the potential to offer speedups and better predictive accuracy compared to their classical counterparts. However, these quantum algorithms, like their classical counterparts, have been shown to also be vulnerable to input perturbations, in particular for classification problems. These can arise either from noisy implementations or, as a worst-case type of noise, adversarial attacks. In order to develop defence mechanisms and to better understand the reliability of these algorithms, it is crucial to understand their robustness properties in presence of natur…
UK Frontier AI Taskforce（英国前沿人工智能工作组）；UC Berkeley（加州大学伯克利分校）；University of Chicago（芝加哥大学）；Center for AI Safety（人工智能安全中心）；Toronto University（多伦多大学）；Vector Institute（向量研究所）
Know-Center GmbH（知识中心有限公司）；Institute of Interactive Systems and Data Science（交互系统与数据科学研究所）；Graz University of Technology（格拉茨工业大学）；Department of Geography and Regional Science（地理与区域科学系）；University of Graz（格拉茨大学）
Correctly quantifying the robustness of machine learning models is a central aspect in judging their suitability for specific tasks, and ultimately, for generating trust in them. We address the problem of finding the robustness of individual predictions. We show both theoretically and with empirical examples that a method based on counterfactuals that was previously proposed for this is insufficient, as it is not a valid metric for determining the robustness against perturbations that occur ``naturally'', outside specific adversarial attack scenarios. We propose a flexible approach that models…
The growing prevalence of counterfeit stories on the internet has fostered significant interest towards fast and scalable detection of fake news in the machine learning community. While several machine learning techniques for this purpose have emerged, we observe that there is a need to evaluate the impact of noise on these techniques' performance, where noise constitutes news articles being mistakenly labeled as fake (or real). This work takes a step in that direction, where we examine the impact of noise on a state-of-the-art, structural model based on BiLSTM (Bidirectional Long-Short Term M…
The growing complexity of Cyber-Physical Systems (CPS) and challenges in ensuring safety and security have led to the increasing use of deep learning methods for accurate and scalable anomaly detection. However, machine learning (ML) models often suffer from low performance in predicting unexpected data and are vulnerable to accidental or malicious perturbations. Although robustness testing of deep learning models has been extensively explored in applications such as image classification and speech recognition, less attention has been paid to ML-driven safety monitoring in CPS. This paper pres…
Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Amsterdam（阿姆斯特丹大学）；Eindhoven University of Technology（埃因霍温理工大学）
Despite their success, modern language models are fragile. Even small changes in their training pipeline can lead to unexpected results. We study this phenomenon by examining the robustness of ALBERT (arXiv:1909.11942) in combination with Stochastic Weight Averaging (SWA) (arXiv:1803.05407) -- a cheap way of ensembling -- on a sentiment analysis task (SST-2). In particular, we analyze SWA's stability via CheckList criteria (arXiv:2005.04118), examining the agreement on errors made by models differing only in their random seed. We hypothesize that SWA is more stable because it ensembles model s…
University College London（伦敦大学学院）；Facebook London（Facebook伦敦）；Kyushu University（九州大学）；Nanyang Technological University（南洋理工大学）
This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing.
Federal Office for Information Security（联邦信息安全办公室）；LatticeFlow
In the last years, AI systems, in particular neural networks, have seen a tremendous increase in performance, and they are now used in a broad range of applications. Unlike classical symbolic AI systems, neural networks are trained using large data sets and their inner structure containing possibly billions of parameters does not lend itself to human interpretation. As a consequence, it is so far not feasible to provide broad guarantees for the correct behaviour of neural networks during operation if they process input data that significantly differ from those seen during training. However, ma…
Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness
European Commission, Joint Research Centre（欧盟委员会联合研究中心）；University of Alcalá（阿尔卡拉大学）；EPFL - Swiss Federal Institute of Technology（瑞士洛桑联邦理工学院）；University of Oxford（牛津大学）；BASt - German Federal Highway Research Institute（德国联邦公路研究所）；Reed Mobility（里德移动公司）
While early empirical evidence has supported the case for learned index structures as having favourable average-case performance, little is known about their worst-case performance. By contrast, classical structures are known to achieve optimal worst-case behaviour. This work evaluates the robustness of learned index structures in the presence of adversarial workloads. To simulate adversarial workloads, we carry out a data poisoning attack on linear regression models that manipulates the cumulative distribution function (CDF) on which the learned index model is trained. The attack deteriorates…
CSI-4CAST: A Hybrid Deep Learning Model for CSI Prediction with Comprehensive Robustness and Generalization Testing
H. Milton Stewart School of Industrial and Systems Engineering, NSF AI Institute for Advances in Optimization (AI4OPT), Georgia Institute of Technology（H.米尔顿·斯图尔特工业与系统工程学院，国家科学基金会人工智能优化先进研究所（AI4OPT），佐治亚理工学院）
Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡国家眼科中心）；Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工学院研究与技术联盟）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Duke-NUS Graduate Medical School（杜克-新加坡国立大学研究生医学院）；National University of Singapore（新加坡国立大学）；Massachusetts Institute of Technology（麻省理工学院）；Institute for Molecular and Clinical Ophthalmology, Basel（巴塞尔分子与临床眼科研究所）；Abyss Processing Pte Ltd（深渊处理私人有限公司）
$\mathbf{Purpose}$: To use artificial intelligence (AI) to: (1) exploit biomechanical knowledge of the optic nerve head (ONH) from a relatively large population; (2) assess ONH robustness from a single optical coherence tomography (OCT) scan of the ONH; (3) identify what critical three-dimensional (3D) structural features make a given ONH robust. $\mathbf{Design}$: Retrospective cross-sectional study. $\mathbf{Methods}$: 316 subjects had their ONHs imaged with OCT before and after acute intraocular pressure (IOP) elevation through ophthalmo-dynamometry. IOP-induced lamina-cribrosa deformations…
In this paper, we examine the overfitting behavior of image classification models modified with Implicit Background Estimation (SCrIBE), which transforms them into weakly supervised segmentation models that provide spatial domain visualizations without affecting performance. Using the segmentation masks, we derive an overfit detection criterion that does not require testing labels. In addition, we assess the change in model performance, calibration, and segmentation masks after applying data augmentations as overfitting reduction measures and testing on various types of distorted images.
Mate! Are You Really Aware? An Explainability-Guided Testing Framework for Robustness of Malware Detectors
CSIRO’s Data61（澳大利亚联邦科学与工业研究组织数据61研究所）；Cybersecurity CRC（网络安全合作研究中心）；Hong Kong University of Science and Technology (GZ)（香港科技大学（广州））；Shanghai Jiao Tong University（上海交通大学）；Peng Cheng Laboratory（鹏城实验室）
Testing practices within the machine learning (ML) community have centered around assessing a learned model's predictive performance measured against a test dataset, often drawn from the same distribution as the training dataset. While recent work on robustness and fairness testing within the ML community has pointed to the importance of testing against distributional shifts, these efforts also focus on estimating the likelihood of the model making an error against a reference dataset/distribution. We argue that this view of testing actively discourages researchers and developers from looking …
We introduce several new datasets namely ImageNet-A/O and ImageNet-R as well as a synthetic environment and testing suite we called CAOS. ImageNet-A/O allow researchers to focus in on the blind spots remaining in ImageNet. ImageNet-R was specifically created with the intention of tracking robust representation as the representations are no longer simply natural but include artistic, and other renditions. The CAOS suite is built off of CARLA simulator which allows for the inclusion of anomalous objects and can create reproducible synthetic environment and scenes for testing robustness. All of t…
University of Pennsylvania（宾夕法尼亚大学）；Baidu Research, Baidu Inc.（百度公司百度研究院）；Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）
Training images with data transformations have been suggested as contrastive examples to complement the testing set for generalization performance evaluation of deep neural networks (DNNs). In this work, we propose a practical framework ContRE (The word "contre" means "against" or "versus" in French.) that uses Contrastive examples for DNN geneRalization performance Estimation. Specifically, ContRE follows the assumption in contrastive learning that robust DNN models with good generalization performance are capable of extracting a consistent set of features and making consistent predictions fr…
Contextual StereoSet: Stress-Testing Bias Alignment Robustness in Large Language Models
Indian Institute of Information Technology, Allahabad (IIITA)（印度信息与技术研究所（Allahabad））；National Institute of Electronics and Information Technology (NIELIT)（国家电子与信息技术研究所）
RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors
University of Cagliari（卡利里大学）；Ruhr University Bochum（波恩鲁尔大学）；University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）；University of Pisa（比萨大学）；Sapienza University of Rome（罗马萨皮恩扎大学）；CINI（CINI研究所）
本文提出 RAID 数据集，包含 7.2 万个高可迁移对抗样本，用于快速、可靠地评估 AI 生成图像检测器的对抗鲁棒性，并揭示现有检测器易受欺骗。
