Reliably predicting potential failure risks of machine learning (ML) systems when deployed with production data is a crucial aspect of trustworthy AI. This paper introduces Risk Advisor, a novel post-hoc meta-learner for estimating failure risks and predictive uncertainties of any already-trained black-box classification model. In addition to providing a risk score, the Risk Advisor decomposes the uncertainty estimates into aleatoric and epistemic uncertainty components, thus giving informative insights into the sources of uncertainty inducing the failures. Consequently, Risk Advisor can distinguish between failures caused by data variability, data shifts and model limitations and advise on mitigation actions (e.g., collecting more data to counter data shift). Extensive experiments on various families of black-box classification models and on real-world and synthetic datasets covering common ML failure scenarios show that the Risk Advisor reliably predicts deployment-time failure risks in all the scenarios, and outperforms strong baselines.
Recent work has shown that deep generative models can assign higher likelihood to out-of-distribution data sets than to their training data (Nalisnick et al., 2019; Choi et al., 2019). We posit that this phenomenon is caused by a mismatch between the model's typical set and its areas of high probability density. In-distribution inputs should reside in the former but not necessarily in the latter, as previous work has presumed. To determine whether or not inputs reside in the typical set, we propose a statistically principled, easy-to-implement test using the empirical distribution of model lik…
Nanyang Technological University（南洋理工大学）；Nanjing University（南京大学）；Chongqing University（重庆大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Detecting out-of-distribution inputs is critical for safe deployment of machine learning models in the real world. However, neural networks are known to suffer from the overconfidence issue, where they produce abnormally high confidence for both in- and out-of-distribution inputs. In this work, we show that this issue can be mitigated through Logit Normalization (LogitNorm) -- a simple fix to the cross-entropy loss -- by enforcing a constant vector norm on the logits in training. Our method is motivated by the analysis that the norm of the logit keeps increasing during training, leading to ove…
INRIA（法国国家信息与自动化研究所）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Beijing Jiaotong University（北京交通大学）；University of Notre Dame（圣母大学）
Machine Learning-as-a-Service systems (MLaaS) have been largely developed for cybersecurity-critical applications, such as detecting network intrusions and fake news campaigns. Despite effectiveness, their robustness against adversarial attacks is one of the key trust concerns for MLaaS deployment. We are thus motivated to assess the adversarial robustness of the Machine Learning models residing at the core of these security-critical applications with categorical inputs. Previous research efforts on accessing model robustness against manipulation of categorical inputs are specific to use cases…
Carnegie Mellon University（卡内基梅隆大学）；University of Pennsylvania（宾夕法尼亚大学）
Recent empirical and theoretical analyses of several commonly used prediction procedures reveal a peculiar risk behavior in high dimensions, referred to as double/multiple descent, in which the asymptotic risk is a non-monotonic function of the limiting aspect ratio of the number of features or parameters to the sample size. To mitigate this undesirable behavior, we develop a general framework for risk monotonization based on cross-validation that takes as input a generic prediction procedure and returns a modified procedure whose out-of-sample prediction risk is, asymptotically, monotonic in …
The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systems
Berkeley AI Safety Initiative (BASIS) UC Berkeley（伯克利人工智能安全计划（BASIS）伯克利大学）；Department of Electrical and Computer Engineering Johns Hopkins University（电气与计算机工程系约翰霍普金斯大学）
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；The Pennsylvania State University（宾夕法尼亚州立大学）
The black-box nature of deep neural networks (DNNs) facilitates attackers to manipulate the behavior of DNN through data poisoning. Being able to detect and mitigate poisoning attacks, typically categorized into backdoor and adversarial poisoning (AP), is critical in enabling safe adoption of DNNs in many application domains. Although recent works demonstrate encouraging results on detection of certain backdoor attacks, they exhibit inherent limitations which may significantly constrain the applicability. Indeed, no technique exists for detecting AP attacks, which represents a harder challenge…
Distributed deep learning frameworks such as split learning provide great benefits with regards to the computational cost of training deep neural networks and the privacy-aware utilization of the collective data of a group of data-holders. Split learning, in particular, achieves this goal by dividing a neural network between a client and a server so that the client computes the initial set of layers, and the server computes the rest. However, this method introduces a unique attack vector for a malicious server attempting to steal the client's private data: the server can direct the client mode…
Detecting and Mitigating Mode-Collapse for Flow-based Sampling of Lattice Field Theories
University of Bonn（波恩大学）；Helmholtz Institute for Radiation and Nuclear Physics (HISKP)（亥姆霍兹辐射与核物理研究所）；Berlin Institute for the Foundations of Learning and Data (BIFOLD)（柏林学习与数据基础研究所）；Technische Universität Berlin（柏林工业大学）；Northeastern University - London（东北大学伦敦校区）；Deutsches Elektronen-Synchrotron DESY（德国电子同步加速器研究所）；Roche（罗氏公司）；RIKEN Center for AIP（理化学研究所先进智能研究中心）
Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation
University of California, Los Angeles（加州大学洛杉矶分校）；The University of Queensland（昆士兰大学）；vivo Mobile Communication Co., Ltd.（vivo移动通信有限公司）；University of California, Merced（加州大学默塞德分校）
DMGNN: Detecting and Mitigating Backdoor Attacks in Graph Neural Networks
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Yangzhou University（扬州大学）；University of Southern Queensland（南昆士兰大学）；Data61, CSIRO（联邦科学与工业研究组织Data61）；University of Technology Sydney（悉尼科技大学）
Detecting Dataset Bias in Medical AI: A Generalized and Modality-Agnostic Auditing Framework
The Johns Hopkins University（约翰霍普金斯大学）；The Johns Hopkins University Applied Physics Laboratory（约翰霍普金斯大学应用物理实验室）；Center for Devices and Radiological Health, U.S. Food and Drug Administration（美国食品药品监督管理局设备与放射健康中心）
Reliably detecting diseases using relevant biological information is crucial for real-world applicability of deep learning techniques in medical imaging. We debias deep learning models during training against unknown bias - without preprocessing/filtering the input beforehand or assuming specific knowledge about its distribution or precise nature in the dataset. We use control regions as surrogates that carry information regarding the bias, employ the classifier model to extract features, and suppress biased intermediate features with our custom, modular DecorreLayer. We evaluate our method on…
University of Virginia（弗吉尼亚大学）；University of Georgia（佐治亚大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
Graph Neural Networks (GNNs) have shown superior performance in analyzing attributed networks in various web-based applications such as social recommendation and web search. Nevertheless, in high-stake decision-making scenarios such as online fraud detection, there is an increasing societal concern that GNNs could make discriminatory decisions towards certain demographic groups. Despite recent explorations on fair GNNs, these works are tailored for a specific GNN model. However, myriads of GNN variants have been proposed for different applications, and it is costly to fine-tune existing debias…
University of Liechtenstein（列支敦士登大学）；Northwestern University（西北大学）
Although machine learning based algorithms have been extensively used for detecting phishing websites, there has been relatively little work on how adversaries may attack such "phishing detectors" (PDs for short). In this paper, we propose a set of Gray-Box attacks on PDs that an adversary may use which vary depending on the knowledge that he has about the PD. We show that these attacks severely degrade the effectiveness of several existing PDs. We then propose the concept of operation chains that iteratively map an original set of features to a new set of features and develop the "Protective …
