Deep networks often make confident, yet, incorrect, predictions when tested with outlier data that is far removed from their training distributions. Likelihoods computed by deep generative models (DGMs) are a candidate metric for outlier detection with unlabeled data. Yet, previous studies have shown that DGM likelihoods are unreliable and can be easily biased by simple transformations to input data. Here, we examine outlier detection with variational autoencoders (VAEs), among the simplest of DGMs. We propose novel analytical and algorithmic approaches to ameliorate key biases with VAE likelihoods. Our bias corrections are sample-specific, computationally inexpensive, and readily computed for various decoder visible distributions. Next, we show that a well-known image pre-processing technique -- contrast stretching -- extends the effectiveness of bias correction to further improve outlier detection. Our approach achieves state-of-the-art accuracies with nine grayscale and natural image datasets, and demonstrates significant advantages -- both with speed and performance -- over four recent, competing approaches. In summary, lightweight remedies suffice to achieve robust outlier detection with VAEs.
University of Edinburgh（爱丁堡大学）；Amazon Development Centre Scotland（亚马逊苏格兰开发中心）；Google AI（谷歌人工智能）；Alan Turing Institute（艾伦·图灵研究所）
Data cleaning often comprises outlier detection and data repair. Systematic errors result from nearly deterministic transformations that occur repeatedly in the data, e.g. specific image pixels being set to default values or watermarks. Consequently, models with enough capacity easily overfit to these errors, making detection and repair difficult. Seeing as a systematic outlier is a combination of patterns of a clean instance and systematic error patterns, our main insight is that inliers can be modelled by a smaller representation (subspace) in a model than outliers. By exploiting this, we pr…
Image generative models can learn the distributions of the training data and consequently generate examples by sampling from these distributions. However, when the training dataset is corrupted with outliers, generative models will likely produce examples that are also similar to the outliers. In fact, a small portion of outliers may induce state-of-the-art generative models, such as Vector Quantized-Variational AutoEncoder (VQ-VAE), to learn a significant mode from the outliers. To mitigate this problem, we propose a robust generative model based on VQ-VAE, which we name Robust VQ-VAE (RVQ-VA…
University of Southern California（南加州大学）；Stevens Institute of Technology（史蒂文斯理工学院）
Machine learning methods often need a large amount of labeled training data. Since the training data is assumed to be the ground truth, outliers can severely degrade learned representations and performance of trained models. Here we apply concepts from robust statistics to derive a novel variational autoencoder that is robust to outliers in the training data. Variational autoencoders (VAEs) extract a lower-dimensional encoded feature representation from which we can generate new data samples. Robustness of autoencoders to outliers is critical for generating a reliable representation of particu…
Variational autoencoders (VAEs) and other generative methods have garnered growing interest not just for their generative properties but also for the ability to dis-entangle a low-dimensional latent variable space. However, few existing generative models take causality into account. We propose a new decoder based framework named the Causal Counterfactual Generative Model (CCGM), which includes a partially trainable causal layer in which a part of a causal model can be learned without significantly impacting reconstruction fidelity. By learning the causal relationships between image semantic la…
University of Southern California（南加利福尼亚大学）；Amazon Web Service（亚马逊云科技）
We propose a robust variational autoencoder with $β$ divergence for tabular data (RTVAE) with mixed categorical and continuous features. Variational autoencoders (VAE) and their variations are popular frameworks for anomaly detection problems. The primary assumption is that we can learn representations for normal patterns via VAEs and any deviation from that can indicate anomalies. However, the training data itself can contain outliers. The source of outliers in training data include the data collection process itself (random noise) or a malicious attacker (data poisoning) who may target to de…
In anomaly detection (AD), one seeks to identify whether a test sample is abnormal, given a data set of normal samples. A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs), for unsupervised learning of the normal data distribution. In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies. In this work, we propose two variational methods for training VAEs for SSAD. The intuitive idea in both methods is to train the encoder to `separate' between latent vectors for normal and outlier data. We show that thi…
Variation Autoencoder (VAE) has become a powerful tool in modeling the non-linear generative process of data from a low-dimensional latent space. Recently, several studies have proposed to use VAE for unsupervised clustering by using mixture models to capture the multi-modal structure of latent representations. This strategy, however, is ineffective when there are outlier data samples whose latent representations are meaningless, yet contaminating the estimation of key major clusters in the latent space. This exact problem arises in the context of resting-state fMRI (rs-fMRI) analysis, where c…
University of Minnesota（明尼苏达大学）；Duke Kunshan University（昆山杜克大学）
We propose a new method for novelty detection that can tolerate high corruption of the training points, whereas previous works assumed either no or very low corruption. Our method trains a robust variational autoencoder (VAE), which aims to generate a model for the uncorrupted training points. To gain robustness to high corruption, we incorporate the following four changes to the common VAE: 1. Extracting crucial features of the latent code by a carefully designed dimension reduction component for distributions; 2. Modeling the latent distribution as a mixture of Gaussian low-rank inliers and …
University of Southern California（南加利福尼亚大学）；Amazon Web Services（亚马逊云科技）
Despite impressive state-of-the-art performance on a wide variety of machine learning tasks, deep learning methods can produce over-confident predictions, particularly with limited training data. Therefore, quantifying uncertainty is particularly important in critical applications such as lesion detection and clinical diagnosis, where a realistic assessment of uncertainty is essential in determining surgical margins, disease status and appropriate treatment. In this work, we propose a novel approach that uses quantile regression for quantifying aleatoric uncertainty in both supervised and unsu…
Amazon Web Services（亚马逊网络服务公司）；University of Southern California（南加州大学）；Los Angeles, CA, USA（美国加利福尼亚州洛杉矶）
Estimation of uncertainty in deep learning models is of vital importance, especially in medical imaging, where reliance on inference without taking into account uncertainty could lead to misdiagnosis. Recently, the probabilistic Variational AutoEncoder (VAE) has become a popular model for anomaly detection in applications such as lesion detection in medical images. The VAE is a generative graphical model that is used to learn the data distribution from samples and then generate new samples from this distribution. By training on normal samples, the VAE can be used to detect inputs that deviate …
Institute for Advanced Study（高等研究院）；Tsinghua University（清华大学）；University of Cambridge（剑桥大学）；Microsoft Research（微软研究院）
Variational autoencoders (VAE) represent a popular, flexible form of deep generative model that can be stochastically fit to samples from a given random process using an information-theoretic variational bound on the true underlying distribution. Once so-obtained, the model can be putatively used to generate new samples from this distribution, or to provide a low-dimensional latent representation of existing samples. While quite effective in numerous application domains, certain important mechanisms which govern the behavior of the VAE are obfuscated by the intractable integrals and resulting …
Deakin University（迪肯大学）；Japan Advanced Institute of Science and Technology（日本北陆先端科学技术大学院大学）
We propose a framework called HyperVAE for encoding distributions of distributions. When a target distribution is modeled by a VAE, its neural network parameters θis drawn from a distribution p(θ) which is modeled by a hyper-level VAE. We propose a variational inference using Gaussian mixture models to implicitly encode the parameters θinto a low dimensional Gaussian distribution. Given a target distribution, we predict the posterior distribution of the latent code, then use a matrix-network decoder to generate a posterior distribution q(θ). HyperVAE can encode the parameters θin full in contr…
Pro-t-VAE: Closed-Form Student's t Mixtures for Robust Prototype Discovery
Institute of Intelligent Information Processing, Shanxi University（山西大学智能信息处理研究所）；School of Medicine, Yale University（耶鲁大学医学院）；Boston Children’s Hospital, Harvard Medical School（哈佛医学院波士顿儿童医院）；Eli Lilly and Company（礼来公司）
Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders
Department of Computer Science and Engineering, Washington University in St. Louis, USA（计算机科学与工程系，华盛顿大学圣路易斯分校）；Institute for Informatics, Data Science and Biostatistics, Washington University in St.Louis, USA（信息学、数据科学与生物统计研究所，华盛顿大学圣路易斯分校）；Department of Radiology, Washington University in St.Louis, USA（放射学系，华盛顿大学圣路易斯分校）
Romanian Institute of Science and Technology（罗马尼亚科学技术研究所）；Babeş-Bolyai University（巴贝什-博尧依大学）
Anomaly detection for Magnetic Resonance Images (MRIs) can be solved with unsupervised methods by learning the distribution of healthy images and identifying anomalies as outliers. In presence of an additional dataset of unlabelled data containing also anomalies, the task can be framed as a semi-supervised task with negative and unlabelled sample points. Recently, in Albu et al., 2020, we have proposed a slice-wise semi-supervised method for tumour detection based on the computation of a dissimilarity function in the latent space of a Variational AutoEncoder, trained on unlabelled data. The di…
Crafting Imperceptible On-Manifold Adversarial Attacks for Tabular Data
School of Information Systems, Queensland University of Technology（信息系统学院，昆士兰技术大学）；Research Centre for Information Systems Engineering (LIRIS), KU Leuven（信息系统工程研究中心（LIRIS），鲁汶大学）；Data Science Institute, University of Technology（数据科学研究院，技术大学）；INESC-ID/Instituto Superior Técnico, University of Lisboa（INESC-ID/理工学院，里斯本大学）
Learning novel representations of variable sources from multi-modal $\textit{Gaia}$ data via autoencoders
Institute of Astronomy, KU Leuven, Celestijnenlaan 200D, B-3001 Leuven, Belgium Millennium Institute of Astrophysics, Nuncio Monse\ nor Sotero Sanz 100, Of. 104, Providencia, Santiago, Chile Department of Astronomy, University of Geneva, Chemin Pegasi 51, 1290 Versoix, Switzerland Department of Astronomy, University of Geneva, Chemin d’Ecogia 16, 1290 Versoix, Switzerland Sednai S\`arl, Geneva, Switzerland INAF - Osservatorio Astrofisico di Torino, Via Osservatorio 20, I-10025 Pino Torinese, Italy Konkoly Observatory, HUN-REN Research Centre for Astronomy；Earth Sciences, Konkoly Thege 15-17, 1121 Budapest, Hungary CSFK, MTA Centre of Excellence, Konkoly Thege 15-17, 1121, Budapest, Hungary INAF - Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, Bologna 40129, Italy Starion for European Space Agency, Camino bajo del Castillo, s/n, Urbanizacion Villafranca del Castillo, Villanueva de la Ca \ n ada, 28692 Madrid, Spain Department of Astrophysics, IMAPP, Radboud University Nijmegen, PO Box 9010, 6500 GL Nijmegen, The Netherlands Max Planck Institute for Astronomy, Koenigstuhl 17, 69117 Heidelberg, Germany
