Training and using modern neural-network based latent-variable generative models (like Variational Autoencoders) often require simultaneously training a generative direction along with an inferential(encoding) direction, which approximates the posterior distribution over the latent variables. Thus, the question arises: how complex does the inferential model need to be, in order to be able to accurately model the posterior distribution of a given generative model?
In this paper, we identify an important property of the generative map impacting the required size of the encoder. We show that if the generative map is "strongly invertible" (in a sense we suitably formalize), the inferential model need not be much more complex. Conversely, we prove that there exist non-invertible generative maps, for which the encoding direction needs to be exponentially larger (under standard assumptions in computational complexity). Importantly, we do not require the generative model to be layerwise invertible, which a lot of the related literature assumes and isn't satisfied by many architectures used in practice (e.g. convolution and pooling based networks). Thus, we provide theoretical support for the empirical wisdom that learning deep generative models is harder when data lies on a low-dimensional manifold.
School of Computing（计算机学院）；University of Kent（肯特大学）
The ability of Variational Autoencoders (VAEs) to learn disentangled representations has made them popular for practical applications. However, their behaviour is not yet fully understood. For example, the questions of when they can provide disentangled representations, or suffer from posterior collapse are still areas of active research. Despite this, there are no layerwise comparisons of the representations learned by VAEs, which would further our understanding of these models. In this paper, we thus look into the internal behaviour of VAEs using representational similarity techniques. Speci…
Invertible flow-based generative models are an effective method for learning to generate samples, while allowing for tractable likelihood computation and inference. However, the invertibility requirement restricts models to have the same latent dimensionality as the inputs. This imposes significant architectural, memory, and computational costs, making them more challenging to scale than other classes of generative models such as Variational Autoencoders (VAEs). We propose a generative model based on probability flows that does away with the bijectivity requirement on the model and only assume…
Samsung AI Center（三星人工智能中心）；Rutgers University（罗格斯大学）
Inference networks of traditional Variational Autoencoders (VAEs) are typically amortized, resulting in relatively inaccurate posterior approximation compared to instance-wise variational optimization. Recent semi-amortized approaches were proposed to address this drawback; however, their iterative gradient update procedures can be computationally demanding. To address these issues, in this paper we introduce an accurate amortized inference algorithm. We propose a novel recursive mixture estimation algorithm for VAEs that iteratively augments the current mixture with new components so as to ma…
Variational AutoEncoders (VAEs) provide a means to generate representational latent embeddings. Previous research has highlighted the benefits of achieving representations that are disentangled, particularly for downstream tasks. However, there is some debate about how to encourage disentanglement with VAEs and evidence indicates that existing implementations of VAEs do not achieve disentanglement consistently. The evaluation of how well a VAE's latent space has been disentangled is often evaluated against our subjective expectations of which attributes should be disentangled for a given probl…
Dr. Ing. h.c. F. Porsche AG（保时捷股份公司）；Ludwig Maximilian University（路德维希-马克西米利安大学）；University of Tübingen（蒂宾根大学）；Chemnitz University of Technology（开姆尼茨工业大学）；Silicon Austria Labs（奥地利硅谷实验室）
There is a growing need for sparse representational formats of human affective states that can be utilized in scenarios with limited computational memory resources. We explore whether representing neural data, in response to emotional stimuli, in a latent vector space can serve to both predict emotional states as well as generate synthetic EEG data that are participant- and/or emotion-specific. We propose a conditional variational autoencoder based framework, EEG2Vec, to learn generative-discriminative representations from EEG data. Experimental results on affective EEG recording datasets demo…
Posterior Collapse as a Phase Transition in Variational Autoencoders
Department of Complexity Science and Engineering, Graduate School of Frontier Sciences, The University of Tokyo（复杂科学与工程系，前沿科学研究生学校，东京大学）；Department of Human and Engineered Environmental Studies, Graduate School of Frontier Sciences, The University of Tokyo（人类与工程环境研究系，前沿科学研究生学校，东京大学）
Shanghai Jiao Tong University（上海交通大学）；University of Michigan-Shanghai Jiao Tong University Joint Institute（密歇根大学-上海交通大学联合学院）；School of Cyber Science and Engineering（网络空间科学与工程学院）；School of Software（软件学院）
Sequential recommendation as an emerging topic has attracted increasing attention due to its important practical significance. Models based on deep learning and attention mechanism have achieved good performance in sequential recommendation. Recently, the generative models based on Variational Autoencoder (VAE) have shown the unique advantage in collaborative filtering. In particular, the sequential VAE model as a recurrent version of VAE can effectively capture temporal dependencies among items in user sequence and perform sequential recommendation. However, VAE-based models suffer from a com…
Department of Computer Science University of Quebec at Montreal（魁北克大学蒙特利尔分校计算机科学系）；University of Quebec at Montreal（魁北克大学蒙特利尔分校）
While representation learning has yielded a great success on many graph learning tasks, there is little understanding behind the structures that are being captured by these embeddings. For example, we wonder if the topological features, such as the Triangle Count, the Degree of the node and other centrality measures are concretely encoded in the embeddings. Furthermore, we ask if the presence of these structures in the embeddings is necessary for a better performance on the downstream tasks, such as clustering and classification. To address these questions, we conduct an extensive empirical st…
In this paper we explore techniques for generating new music using a Variational Autoencoder (VAE) neural network that was trained on a corpus of specific style. Instead of randomly sampling the latent states of the network to produce free improvisation, we generate new music by querying the network with musical input in a style different from the training corpus. This allows us to produce new musical output with longer-term structure that blends aspects of the query to the style of the network. In order to control the level of this blending we add a noisy channel between the VAE encoder and d…
Autoencoder networks are unsupervised approaches aiming at combining generative and representational properties by learning simultaneously an encoder-generator map. Although studied extensively, the issues of whether they have the same generative power of GANs, or learn disentangled representations, have not been fully addressed. We introduce an autoencoder that tackles these issues jointly, which we call Adversarial Latent Autoencoder (ALAE). It is a general architecture that can leverage recent improvements on GAN training procedures. We designed two autoencoders: one based on a MLP encoder,…
Australian National University（澳大利亚国立大学）；Data61-CSIRO（澳大利亚联邦科学与工业研究组织数据61中心）
Ever since the advent of AlexNet, designing novel deep neural architectures for different tasks has consistently been a productive research direction. Despite the exceptional performance of various architectures in practice, we study a theoretical question: what is the condition for deep neural architectures to preserve all the information of the input data? Identifying the information lossless condition for deep neural architectures is important, because tasks such as image restoration require keep the detailed information of the input data as much as possible. Using the definition of mutual …
Mechanical and Aerospace Engineering, Princeton University（普林斯顿大学机械与航空航天工程系（注：此处按要求提取到大学级别，故保留系名与大学名））；Princeton University（普林斯顿大学）
This paper describes a method for learning low-dimensional approximations of nonlinear dynamical systems, based on neural-network approximations of the underlying Koopman operator. Extended Dynamic Mode Decomposition (EDMD) provides a useful data-driven approximation of the Koopman operator for analyzing dynamical systems. This paper addresses a fundamental problem associated with EDMD: a trade-off between representational capacity of the dictionary and over-fitting due to insufficient data. A new neural network architecture combining an autoencoder with linear recurrent dynamics in the encode…
The recently developed variational autoencoders (VAEs) have proved to be an effective confluence of the rich representational power of neural networks with Bayesian methods. However, most work on VAEs use a rather simple prior over the latent variables such as standard normal distribution, thereby restricting its applications to relatively simple phenomena. In this work, we propose hierarchical nonparametric variational autoencoders, which combines tree-structured Bayesian nonparametric priors with VAEs, to enable infinite flexibility of the latent representation space. Both the neural paramet…
Establishing correspondence between images or scenes is a significant challenge in computer vision, especially given occlusions, viewpoint changes, and varying object appearances. In this paper, we present Siamese Masked Autoencoders (SiamMAE), a simple extension of Masked Autoencoders (MAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them. These frames are processed independently by an encoder network, and a decoder composed of a sequence of cross-attention layers is tasked with predicting the missing patc…
FFHFlow: Diverse and Uncertainty-Aware Dexterous Grasp Generation via Flow Variational Inference
School of Information Computation and Technology, Technical University of Munich (TUM)（技术大学慕尼黑信息计算技术学院）；Institute of Robotics and Mechatronics, German Aerospace Center (DLR)（德国航空航天中心机器人与机电研究所）；Department of Informatics, Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院信息学院）
THeGAU: Type-Aware Heterogeneous Graph Autoencoder and Augmentation
Data Science Degree Program National Taiwan University；Graduate Institute of Communication Engineering National Taiwan University Taipei Taiwan；University of Illinois Urbana-Champaign Electrical \& Computer Engineering Illinois United States；Research Center for Information Technology Innovation Academia Sinica Taipei Taiwan；Department of Electrical Engineering National Taiwan University Taipei Taiwan；National Taiwan University；University of Illinois Urbana-Champaign
Biologically Informed Deep Neural Networks for Multi-Omic Integration, Pathway Activity Inference and Risk Stratification in Cancer
King’s College London（伦敦国王学院）；Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（科技研究局信息与通信研究所）；Shenzhen University（深圳大学）；University of Manchester（曼彻斯特大学）
University of California San Diego（加利福尼亚大学圣迭戈分校）；Georgia Institute of Technology（佐治亚理工学院）；University of California Riverside（加利福尼亚大学河滨分校）；San Diego State University（圣迭戈州立大学）
Institute of Neuroinformatics, University of Zurich and ETH Zurich（苏黎世大学与苏黎世联邦理工学院神经信息学研究所）；UCLA（加州大学洛杉矶分校）；Institute for Theoretical Physics, ETH Zurich（苏黎世联邦理工学院理论物理研究所）；Center of Economic Research, ETH Zurich（苏黎世联邦理工学院经济研究中心）
Recent advances in deep learning and neural networks have led to an increased interest in the application of generative models in statistical and condensed matter physics. In particular, restricted Boltzmann machines (RBMs) and variational autoencoders (VAEs) as specific classes of neural networks have been successfully applied in the context of physical feature extraction and representation learning. Despite these successes, however, there is only limited understanding of their representational properties and limitations. To better understand the representational characteristics of RBMs and V…
