A crucial requirement for reliable deployment of deep learning models for safety-critical applications is the ability to identify out-of-distribution (OOD) data points, samples which differ from the training data and on which a model might underperform. Previous work has attempted to tackle this problem using uncertainty estimation techniques. However, there is empirical evidence that a large family of these techniques do not detect OOD reliably in classification tasks.
This paper gives a theoretical explanation for said experimental findings and illustrates it on synthetic data. We prove that such techniques are not able to reliably identify OOD samples in a classification setting, since their level of confidence is generalized to unseen areas of the feature space. This result stems from the interplay between the representation of ReLU networks as piece-wise affine transformations, the saturating nature of activation functions like softmax, and the most widely-used uncertainty metrics.
It has recently been shown that ReLU networks produce arbitrarily over-confident predictions far away from the training data. Thus, ReLU networks do not know when they don't know. However, this is a highly important property in safety critical applications. In the context of out-of-distribution detection (OOD) there have been a number of proposals to mitigate this problem but none of them are able to make any mathematical guarantees. In this paper we propose a new approach to OOD which overcomes both problems. Our approach can be used with ReLU networks and provides provably low confidence pre…
Warsaw University of Technology（华沙理工大学）；Silesian University of Technology（西里西亚工业大学）；NVIDIA（英伟达）
Deep neural networks (DNN) have outstanding performance in various applications. Despite numerous efforts of the research community, out-of-distribution (OOD) samples remain a significant limitation of DNN classifiers. The ability to identify previously unseen inputs as novel is crucial in safety-critical applications such as self-driving cars, unmanned aerial vehicles, and robots. Existing approaches to detect OOD samples treat a DNN as a black box and evaluate the confidence score of the output predictions. Unfortunately, this method frequently fails, because DNNs are not trained to reduce t…
Shanghai Jiao Tong University（上海交通大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California San Diego（加州大学圣地亚哥分校）
This paper aims to theoretically analyze the complexity of feature transformations encoded in piecewise linear DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN's complexity. The proposed metrics can also be used as a loss t…
College of Artificial Intelligence and Automation, Hohai University, China（人工智能与自动化学院，河海大学，中国）；Zhongguancun Laboratory, Beijing, China（中关村实验室，北京，中国）；College of Information Science and Engineering, Hohai University, China（信息科学与工程学院，河海大学，中国）
Out-of-distribution (OOD) learning deals with scenarios in which training and test data follow different distributions. Although general OOD problems have been intensively studied in machine learning, graph OOD is only an emerging area of research. Currently, there lacks a systematic benchmark tailored to graph OOD method evaluation. In this work, we aim at developing an OOD benchmark, known as GOOD, for graphs specifically. We explicitly make distinctions between covariate and concept shifts and design data splits that accurately reflect different shifts. We consider both graph and node predi…
When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
Sydney AI Centre, The University of Sydney（悉尼人工智能中心，悉尼大学）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机系）
Google Research India（谷歌印度研究院）；Indian Institute of Science, Bangalore（印度科学学院班加罗尔分院）
We consider the problem of OOD generalization, where the goal is to train a model that performs well on test distributions that are different from the training distribution. Deep learning models are known to be fragile to such shifts and can suffer large accuracy drops even for slightly different test distributions. We propose a new method - DAFT - based on the intuition that adversarially robust combination of a large number of rich features should provide OOD robustness. Our method carefully distills the knowledge from a powerful teacher that learns several discriminative features using stan…
University of California, San Diego（加利福尼亚大学圣迭戈分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Estimating out-of-distribution (OOD) uncertainty is a central challenge for safely deploying machine learning models in the open-world environment. Improved methods for OOD detection in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose JointEnergy, a simple and effective method, which estimates the OOD indicator scores by aggregating energy scores from multiple labels. We show that JointEnergy can be mathematically interpreted from a joint likelihood perspective. Our results show c…
UC Berkeley（加州大学伯克利分校）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Improving the accuracy of deep neural networks (DNNs) on out-of-distribution (OOD) data is critical to an acceptance of deep learning (DL) in real world applications. It has been observed that accuracies on in-distribution (ID) versus OOD data follow a linear trend and models that outperform this baseline are exceptionally rare (and referred to as "effectively robust"). Recently, some promising approaches have been developed to improve OOD robustness: model pruning, data augmentation, and ensembling or zero-shot evaluating large pretrained models. However, there still is no clear understanding…
Modern deep neural network models are known to erroneously classify out-of-distribution (OOD) test data into one of the in-distribution (ID) training classes with high confidence. This can have disastrous consequences for safety-critical applications. A popular mitigation strategy is to train a separate classifier that can detect such OOD samples at the test time. In most practical settings OOD examples are not known at the train time, and hence a key question is: how to augment the ID data with synthetic OOD samples for training such an OOD detector? In this paper, we propose a novel Compound…
Fudan University（复旦大学）；Beihang University（北京航空航天大学）；Samsung AI Center（三星人工智能中心）
Neural networks are known to produce over-confident predictions on input images, even when these images are out-of-distribution (OOD) samples. This limits the applications of neural network models in real-world scenarios, where OOD samples exist. Many existing approaches identify the OOD instances via exploiting various cues, such as finding irregular patterns in the feature space, logits space, gradient space or the raw space of images. In contrast, this paper proposes a simple Test-time Linear Training (ETLT) method for OOD detection. Empirically, we find that the probabilities of input imag…
GLIP-OOD: Zero-Shot Graph OOD Detection with Graph Foundation Model
University of Southern California（南加州大学）；University of Maryland, College Park（马里兰大学学院公园分校）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Florida State University（佛罗里达州立大学）
By design, discriminatively trained neural network classifiers produce reliable predictions only for in-distribution samples. For their real-world deployments, detecting out-of-distribution (OOD) samples is essential. Assuming OOD to be outside the closed boundary of in-distribution, typical neural classifiers do not contain the knowledge of this boundary for OOD detection during inference. There have been recent approaches to instill this knowledge in classifiers by explicitly training the classifier with OOD samples close to the in-distribution boundary. However, these generated samples fail…
Learning to Augment Distributions for Out-of-Distribution Detection
Hong Kong Baptist University（香港浸会大学）；University of Technology Sydney（悉尼科技大学）；The University of Melbourne（墨尔本大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Can Molecular Foundation Models Know What They Don't Know? A Simple Remedy with Preference Optimization
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）；Technical University of Berlin（柏林工业大学）；University of Southern California（南加州大学）；Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学博德研究所）
University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）
The application of machine learning in safety-critical systems requires a reliable assessment of uncertainty. However, deep neural networks are known to produce highly overconfident predictions on out-of-distribution (OOD) data. Even if trained to be non-confident on OOD data, one can still adversarially manipulate OOD data so that the classifier again assigns high confidence to the manipulated samples. We show that two previously published defenses can be broken by better adapted attacks, highlighting the importance of robustness guarantees around OOD data. Since the existing method for this …
University of Pennsylvania（宾夕法尼亚大学）；SRI International（SRI国际）
Deep neural networks (DNNs) are known to produce incorrect predictions with very high confidence on out-of-distribution inputs (OODs). This limitation is one of the key challenges in the adoption of DNNs in high-assurance systems such as autonomous driving, air traffic management, and medical diagnosis. This challenge has received significant attention recently, and several techniques have been developed to detect inputs where the model's prediction cannot be trusted. These techniques detect OODs as datapoints with either high epistemic uncertainty or high aleatoric uncertainty. We demonstrate…
Tencent AI Lab（腾讯人工智能实验室）；The University of Hong Kong（香港大学）；Fudan University（复旦大学）
AI-aided drug discovery (AIDD) is gaining increasing popularity due to its promise of making the search for new pharmaceuticals quicker, cheaper and more efficient. In spite of its extensive use in many fields, such as ADMET prediction, virtual screening, protein folding and generative chemistry, little has been explored in terms of the out-of-distribution (OOD) learning problem with \emph{noise}, which is inevitable in real world AIDD applications. In this work, we present DrugOOD, a systematic OOD dataset curator and benchmark for AI-aided drug discovery, which comes with an open-source Pyth…
OODD: Test-time Out-of-Distribution Detection with Dynamic Dictionary
Shanghai Jiao Tong University（上海交通大学）；Tianjin University（天津大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
In an out-of-distribution (OOD) detection problem, samples of known classes(also called in-distribution classes) are used to train a special classifier. In testing, the classifier can (1) classify the test samples of known classes to their respective classes and also (2) detect samples that do not belong to any of the known classes (i.e., they belong to some unknown or OOD classes). This paper studies the problem of zero-shot out-of-distribution(OOD) detection, which still performs the same two tasks in testing but has no training except using the given known class names. This paper proposes a…
School of Computer Science and Engineering, Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, Sun Yat-sen University（计算机科学与工程学院，人工智能与先进计算重点实验室，教育部，中山大学）；School of Computing, Macquarie University（计算学院，麦考瑞大学）；Statistics and Data Science, Southeast University（统计与数据科学，东南大学）；School of Computer Science and Engineering, Sun Yat-sen University（计算机科学与工程学院，中山大学）
University of Pennsylvania（宾夕法尼亚大学）；SRI International（SRI国际）
Deep neural networks (DNNs) are known to produce incorrect predictions with very high confidence on out-of-distribution (OOD) inputs. This limitation is one of the key challenges in the adoption of deep learning models in high-assurance systems such as autonomous driving, air traffic management, and medical diagnosis. This challenge has received significant attention recently, and several techniques have been developed to detect inputs where the model's prediction cannot be trusted. These techniques use different statistical, geometric, or topological signatures. This paper presents a taxonomy…
California Institute of Technology（加州理工学院）；Rice University（莱斯大学）
Out-of-distribution (OoD) detection is a natural downstream task for deep generative models, due to their ability to learn the input probability distribution. There are mainly two classes of approaches for OoD detection using deep generative models, viz., based on likelihood measure and the reconstruction loss. However, both approaches are unable to carry out OoD detection effectively, especially when the OoD samples have smaller variance than the training samples. For instance, both flow based and VAE models assign higher likelihood to images from SVHN when trained on CIFAR-10 images. We use …
Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector
University of Manchester（曼彻斯特大学）；University of Tübingen（蒂宾根大学）；University of Cambridge（剑桥大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
PRECISE Lab Computer and Information Science Dept. University of Pennsylvania（宾夕法尼亚大学计算机与信息科学系PRECISE实验室）；Dermatology Dept. University of Pennsylvania（宾夕法尼亚大学皮肤科）
As machine learning models continue to achieve impressive performance across different tasks, the importance of effective anomaly detection for such models has increased as well. It is common knowledge that even well-trained models lose their ability to function effectively on out-of-distribution inputs. Thus, out-of-distribution (OOD) detection has received some attention recently. In the vast majority of cases, it uses the distribution estimated by the training dataset for OOD detection. We demonstrate that the current detectors inherit the biases in the training dataset, unfortunately. This…
Out-of-distribution (OOD) detection, i.e., finding test samples derived from a different distribution than the training set, as well as reasoning about such samples (OOD reasoning), are necessary to ensure the safety of results generated by machine learning models. Recently there have been promising results for OOD detection in the latent space of variational autoencoders (VAEs). However, without disentanglement, VAEs cannot perform OOD reasoning. Disentanglement ensures a one- to-many mapping between generative factors of OOD (e.g., rain in image data) and the latent variables to which they a…
MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification
Dept. of Computer Science The University of Texas at El Paso Texas, USA（计算机科学系 乌德勒支理工大学 美国德克萨斯州）；Dept. of Computer Science University of North Carolina North Carolina, USA（计算机科学系 北卡罗来纳大学 美国北卡罗来纳州）
University of Tübingen（图宾根大学）；Saarland University（萨尔大学）
Classifiers used in the wild, in particular for safety-critical systems, should not only have good generalization properties but also should know when they don't know, in particular make low confidence predictions far away from the training data. We show that ReLU type neural networks which yield a piecewise linear classifier function fail in this regard as they produce almost always high confidence predictions far away from the training data. For bounded domains like images we propose a new robust optimization technique similar to adversarial training which enforces low confidence predictions…
Accurate estimation of aleatoric and epistemic uncertainty is crucial to build safe and reliable systems. Traditional approaches, such as dropout and ensemble methods, estimate uncertainty by sampling probability predictions from different submodels, which leads to slow uncertainty estimation at inference time. Recent works address this drawback by directly predicting parameters of prior distributions over the probability predictions with a neural network. While this approach has demonstrated accurate uncertainty estimation, it requires defining arbitrary target parameters for in-distribution …
Unified Classification and Rejection: A One-versus-All Framework
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
本文提出一对其余统一框架，将 K 类开放集识别建模为仅用已知类训练的 (K+1) 类问题，并结合 OVA 与交叉熵损失，在单一分类器上兼顾闭集分类、OOD 检测和误分类检测。
Duke University（杜克大学）；Air Force Research Laboratory（空军研究实验室）
Many real-world scenarios in which DNN-based recognition systems are deployed have inherently fine-grained attributes (e.g., bird-species recognition, medical image classification). In addition to achieving reliable accuracy, a critical subtask for these models is to detect Out-of-distribution (OOD) inputs. Given the nature of the deployment environment, one may expect such OOD inputs to also be fine-grained w.r.t. the known classes (e.g., a novel bird species), which are thus extremely difficult to identify. Unfortunately, OOD detection in fine-grained scenarios remains largely underexplored.…
Monash University（莫纳什大学）；Arizona State University（亚利桑那州立大学）；Griffith University（格里菲斯大学）
Most existing deep learning models are trained based on the closed-world assumption, where the test data is assumed to be drawn i.i.d. from the same distribution as the training data, known as in-distribution (ID). However, when models are deployed in an open-world scenario, test samples can be out-of-distribution (OOD) and therefore should be handled with caution. To detect such OOD samples drawn from unknown distribution, OOD detection has received increasing attention lately. However, current endeavors mostly focus on grid-structured data and its application for graph-structured data remain…
Deep neural networks (DNNs) perform well at classifying inputs associated with the classes they have been trained on, which are known as in distribution inputs. However, out-of-distribution (OOD) inputs pose a great challenge to DNNs and consequently represent a major risk when DNNs are implemented in safety-critical systems. Extensive research has been performed in the domain of OOD detection. However, current state-of-the-art methods for OOD detection suffer from at least one of the following limitations: (1) increased inference time - this limits existing methods' applicability to many real…
