Distributions are fundamental statistical elements that play essential theoretical and practical roles. The article discusses experiences of training neural networks to classify univariate empirical distributions and to represent them on the two-dimensional latent space forcing disentanglement based on the inputs of cumulative distribution functions (CDF). The latent space representation has been performed using an unsupervised beta variational autoencoder (beta-VAE). It separates distributions of different shapes while overlapping similar ones and empirically realises relationships between distributions that are known theoretically. The synthetic experiment of generated univariate continuous and discrete (Bernoulli) distributions with varying sample sizes and parameters has been performed to support the study. The representation on the latent two-dimensional coordinate system can be seen as an additional metadata of the real-world data that disentangles important distribution characteristics, such as shape of the CDF, classification probabilities of underlying theoretical distributions and their parameters, information entropy, and skewness. Entropy changes, providing an "arrow of time", determine dynamic trajectories along representations of distributions on the latent space. In addition, post beta-VAE unsupervised segmentation of the latent space based on weight-of-evidence (WOE) of posterior versus standard isotopic two-dimensional normal densities has been applied detecting the presence of assignable causes that distinguish exceptional CDF inputs.
University of Cambridge（剑桥大学）；The Alan Turing Institute（艾伦·图灵研究所）
Variational autoencoders (VAEs) are a powerful class of deep generative latent variable model for unsupervised representation learning on high-dimensional data. To ensure computational tractability, VAEs are often implemented with a univariate standard Gaussian prior and a mean-field Gaussian variational posterior distribution. This results in a vector-valued latent variables that are agnostic to the original data structure which might be highly correlated across and within multiple dimensions. We propose a tensor-variate extension to the VAE framework, the tensor-variate Gaussian process prio…
Univariate Conditional Variational Autoencoder for Morphogenic Patterns Design in Frontal Polymerization-Based Manufacturing
National Center for Supercomputing Applications, University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校国家超级计算应用中心）；Beckman Institute for Advanced Science and Technology, University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校贝克曼先进科学技术研究所）；Grainger College of Engineering, University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校格兰杰工程学院）；New York University Abu Dhabi（纽约大学阿布扎比分校）；Kansas State University（堪萨斯州立大学）；Massachusetts Institute of Technology（麻省理工学院）
University of Southern California（南加利福尼亚大学）；Amazon Web Service（亚马逊云科技）
We propose a robust variational autoencoder with $β$ divergence for tabular data (RTVAE) with mixed categorical and continuous features. Variational autoencoders (VAE) and their variations are popular frameworks for anomaly detection problems. The primary assumption is that we can learn representations for normal patterns via VAEs and any deviation from that can indicate anomalies. However, the training data itself can contain outliers. The source of outliers in training data include the data collection process itself (random noise) or a malicious attacker (data poisoning) who may target to de…
MILA（蒙特利尔学习算法研究所）；University of Montreal（蒙特利尔大学）；Element AI（元素人工智能公司）
Normalizing flows and autoregressive models have been successfully combined to produce state-of-the-art results in density estimation, via Masked Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows (IAF). We unify and generalize these approaches, replacing the (conditionally) affine univariate transformations of MAF/IAF with a more general class of invertible univariate transformations expressed as monotonic neural networks. We demonstrate that the proposed neural autoregressive flows (NAF)…
Institut de Mathématiques de Toulouse（图卢兹数学研究所）；University of Toulouse（图卢兹大学）；Liebherr Aerospace Toulouse（利勃海尔航空航天图卢兹公司）；University Toulouse 3（图卢兹第三大学）
In this work, we explore dimensionality reduction techniques for univariate and multivariate time series data. We especially conduct a comparison between wavelet decomposition and convolutional variational autoencoders for dimension reduction. We show that variational autoencoders are a good option for reducing the dimension of high dimensional data like ECG. We make these comparisons on a real world, publicly available, ECG dataset that has lots of variability and use the reconstruction error as the metric. We then explore the robustness of these models with noisy data whether for training or…
School of Computer Science, Tel Aviv University（特拉维夫大学计算机科学学院）；Facebook AI Research (FAIR)（Facebook AI Research (FAIR)（常见译法：脸书人工智能研究院））
In the univariate case, we show that by comparing the individual complexities of univariate cause and effect, one can identify the cause and the effect, without considering their interaction at all. In our framework, complexities are captured by the reconstruction error of an autoencoder that operates on the quantiles of the distribution. Comparing the reconstruction errors of the two autoencoders, one for each variable, is shown to perform surprisingly well on the accepted causality directionality benchmarks. Hence, the decision as to which of the two is the cause and which is the effect may …
Generating power system states that have similar distribution and dependency to the historical ones is essential for the tasks of system planning and security assessment, especially when the historical data is insufficient. In this paper, we described a generative model for load profiles of industrial and commercial customers, based on the conditional variational autoencoder (CVAE) neural network architecture, which is challenging due to the highly variable nature of such profiles. Generated contextual load profiles were conditioned on the month of the year and typical power exchange with the …
University of Southern California（南加州大学）；Stevens Institute of Technology（史蒂文斯理工学院）
Machine learning methods often need a large amount of labeled training data. Since the training data is assumed to be the ground truth, outliers can severely degrade learned representations and performance of trained models. Here we apply concepts from robust statistics to derive a novel variational autoencoder that is robust to outliers in the training data. Variational autoencoders (VAEs) extract a lower-dimensional encoded feature representation from which we can generate new data samples. Robustness of autoencoders to outliers is critical for generating a reliable representation of particu…
A Multi-Level Causal Intervention Framework for Mechanistic Interpretability in Variational Autoencoders
Indian Institute of Technology, Patna（印度理工学院巴特那分校）；Rajarshi School of Management Technology, Varanasi（拉贾尔希管理技术学院瓦拉纳西分校）；Jaypee Institute of Information Technology, Noida（杰皮信息技术学院诺伊达分校）
For planning of power systems and for the calibration of operational tools, it is essential to analyse system performance in a large range of representative scenarios. When the available historical data is limited, generative models are a promising solution, but modelling high-dimensional dependencies is challenging. In this paper, a multivariate load state generating model on the basis of a conditional variational autoencoder (CVAE) neural network is proposed. Going beyond common CVAE implementations, the model includes stochastic variation of output samples under given latent vectors and co-…
$β$-VAE is a follow-up technique to variational autoencoders that proposes special weighting of the KL divergence term in the VAE loss to obtain disentangled representations. Unsupervised learning is known to be brittle even on toy datasets and a meaningful, mathematically precise definition of disentanglement remains difficult to find. Here we investigate the original $β$-VAE paper and add evidence to the results previously obtained indicating its lack of reproducibility. We also further expand the experimentation of the models and include further more complex datasets in the analysis. We als…
Design of experiments (DOE) is playing an essential role in learning and improving a variety of objects and processes. The article discusses the application of unsupervised machine learning to support the pragmatic designs of complex experiments. Complex experiments are characterized by having a large number of factors, mixed-level designs, and may be subject to constraints that eliminate some unfeasible trials for various reasons. Having such attributes, it is very challenging to design pragmatic experiments that are economically, operationally, and timely sound. It means a significant decrea…
Variational AutoEncoders (VAEs) provide a means to generate representational latent embeddings. Previous research has highlighted the benefits of achieving representations that are disentangled, particularly for downstream tasks. However, there is some debate about how to encourage disentanglement with VAEs and evidence indicates that existing implementations of VAEs do not achieve disentanglement consistently. The evaluation of how well a VAE's latent space has been disentangled is often evaluated against our subjective expectations of which attributes should be disentangled for a given probl…
University College London（伦敦大学学院）；Microsoft Research Cambridge（微软剑桥研究院）
While a wide range of interpretable generative procedures for graphs exist, matching observed graph topologies with such procedures and choices for its parameters remains an open problem. Devising generative models that closely reproduce real-world graphs requires domain knowledge and time-consuming simulation. While existing deep learning approaches rely on less manual modelling, they offer little interpretability. This work approaches graph generation (decoding) as the inverse of graph compression (encoding). We show that in a disentanglement-focused deep autoencoding framework, specifically…
University of Copenhagen（哥本哈根大学）；FaunaPhotonics A/S（FaunaPhotonics公司）
While insects are the largest and most diverse group of terrestrial animals, constituting ca. 80% of all known species, they are difficult to study due to their small size and similarity between species. Conventional monitoring techniques depend on time consuming trapping methods and tedious microscope-based work by skilled experts in order to identify the caught insect specimen at species, or even family level. Researchers and policy makers are in urgent need of a scalable monitoring tool in order to conserve biodiversity and secure human food production due to the rapid decline in insect num…
Stabilization Analysis and Mode Recognition of Kerosene Supersonic Combustion: A Deep Learning Approach Based on Res-CNN-beta-VAE
City University of Hong Kong（香港城市大学）；State Key Laboratory of High Temperature Gas Dynamics, Institute of Mechanics, Chinese Academy of Sciences（中国科学院力学研究所高温气体动力学国家重点实验室）
Multimodal Analysis of White Blood Cell Differentiation in Acute Myeloid Leukemia Patients using a β-Variational Autoencoder
Institute of AI for Health, Helmholtz Zentrum München – German Research Center for Environmental Health（慕尼黑亥姆霍兹中心人工智能健康研究所（德国环境健康研究中心））；University of Trento（特伦托大学）；Technical University of Munich（慕尼黑工业大学）；Ludwig-Maximilians University Munich（慕尼黑大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Microsoft Research（微软研究院）；University of Cambridge（剑桥大学）；Google Brain（谷歌大脑）
Recently there has been an increased interest in unsupervised learning of disentangled representations using the Variational Autoencoder (VAE) framework. Most of the existing work has focused largely on modifying the variational cost function to achieve this goal. We first show that these modifications, e.g. beta-VAE, simplify the tendency of variational inference to underfit causing pathological over-pruning and over-orthogonalization of learned components. Second we propose a complementary approach: to modify the probabilistic model with a structured latent prior. This prior allows to discov…
National Institute of Technology, Warangal（瓦朗加尔国家理工学院）
In recent years, applying Deep Learning (DL) techniques emerged as a common practice in the communication system, demonstrating promising results. The present paper proposes a new Convolutional Neural Network (CNN) based Variational Autoencoder (VAE) communication system. The VAE (continuous latent space) based communication systems confer unprecedented improvement in the system performance compared to AE (distributed latent space) and other traditional methods. We have introduced an adjustable hyperparameter beta in the proposed VAE, which is also known as beta-VAE, resulting in extremely dis…
A large number of objectives have been proposed to train latent variable generative models. We show that many of them are Lagrangian dual functions of the same primal optimization problem. The primal problem optimizes the mutual information between latent and visible variables, subject to the constraints of accurately modeling the data distribution and performing correct amortized inference. By choosing to maximize or minimize mutual information, and choosing different Lagrange multipliers, we obtain different objectives including InfoGAN, ALI/BiGAN, ALICE, CycleGAN, beta-VAE, adversarial auto…
ECG Latent Feature Extraction with Autoencoders for Downstream Prediction Tasks
Department of Cardiovascular Medicine, The University of Kansas Medical Center（心血管医学系，堪萨斯医学院）；Department of Electrical Engineering and Computer Science, The University of Kansas（电气工程与计算机科学系，堪萨斯大学）
The slowness principle is a concept inspired by the visual cortex of the brain. It postulates that the underlying generative factors of a quickly varying sensory signal change on a slower time scale. Unsupervised learning of intermediate representations utilizing abundant unlabeled sensory data can be leveraged to perform data-efficient supervised downstream regression. In this paper, we propose a general formulation of slowness for unsupervised representation learning adding a slowness regularization term to the estimate lower bound of the beta-VAE to encourage temporal similarity in observat…
While the beta-VAE family is aiming to find disentangled representations and acquire human-interpretable generative factors, like what an ICA (from the linear domain) does, we propose Full Encoder, a novel unified autoencoder framework as a correspondence to PCA in the non-linear domain. The idea is to train an autoencoder with one latent variable first, then involve more latent variables progressively to refine the reconstruction results. The Full Encoder is also a latent variable predictive model that the latent variables acquired are stable and robust, as they always learn the same represen…
École de Technologie Supérieure, University of Québec（魁北克大学高等技术学院）；CHU Sainte-Justine Research Center（圣贾斯汀儿童医院研究中心）；CHU Sainte-Justine Hospital（圣贾斯汀儿童医院）；University of Montreal（蒙特利尔大学）
When dealing with clinical text classification on a small dataset recent studies have confirmed that a well-tuned multilayer perceptron outperforms other generative classifiers, including deep learning ones. To increase the performance of the neural network classifier, feature selection for the learning representation can effectively be used. However, most feature selection methods only estimate the degree of linear dependency between variables and select the best features based on univariate statistical tests. Furthermore, the sparsity of the feature space involved in the learning representat…
Computer Vision and Learning Systems (CVL), Linköping University, Sweden（计算机视觉与学习系统（CVL），林雪平大学，瑞典）；Department of Science and Technology, Linköping University, Sweden（科学与技术系，林雪平大学，瑞典）
Revisiting VAE for Unsupervised Time Series Anomaly Detection: A Frequency Perspective
University of Chinese Academy of Sciences（中国科学院大学）；Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；Microsoft（微软公司）；Stony Brook University（石溪大学）；Kuaishou Technology（快手科技）；Tsinghua University（清华大学）
Aix Marseille Université（艾克斯-马赛大学）；Université de Toulon（土伦大学）；Institut de Neurosciences de la Timone（蒂莫纳神经科学研究所）；École Centrale de Marseille（马赛中央理工学院）
In neuroscience, understanding inter-individual differences has recently emerged as a major challenge, for which functional magnetic resonance imaging (fMRI) has proven invaluable. For this, neuroscientists rely on basic methods such as univariate linear correlations between single brain features and a score that quantifies either the severity of a disease or the subject's performance in a cognitive task. However, to this date, task-fMRI and resting-state fMRI have been exploited separately for this question, because of the lack of methods to effectively combine them. In this paper, we introdu…
Bharati Vidyapeeth’s College of Engineering（巴拉蒂维迪佩斯工程学院）；IIT Delhi（印度理工学院德里分校）
Traditional adversarial attacks rely upon the perturbations generated by gradients from the network which are generally safeguarded by gradient guided search to provide an adversarial counterpart to the network. In this paper, we propose a novel mechanism of generating adversarial examples where the actual image is not corrupted rather its latent space representation is utilized to tamper with the inherent structure of the image while maintaining the perceptual quality intact and to act as legitimate data samples. As opposed to gradient-based attacks, the latent space poisoning exploits the in…
