深度潜变量生成模型因其能够以优雅的方式结合深度学习与概率模型的优势而受到越来越多的关注。通过这些模型学习到的数据表示通常是连续且密集的。然而，在许多应用中，人们期望得到稀疏表示，例如在无监督设置中学习数据的高维稀疏嵌入，以及在监督设置中从数千个候选标签中学习多标签。在某些场景下，对稀疏度可能有进一步的限制：表示的非零特征数量不能超过预定义的阈值 $L_0$。在本文中，我们提出了一种稀疏深度潜变量生成模型 SDLGM，以显式地对稀疏度进行建模，从而能够在量化的稀疏约束下学习数据的稀疏结构。表示的最终稀疏度不是固定的，而是在预定义的限制下根据观测数据本身进行自适应拟合。具体而言，我们为每个观测 $i$ 引入一个辅助随机变量 $L_i$，用于对其表示的稀疏度进行建模。随后，通过两个 Gumbel-Softmax 分布，采用两步采样过程生成稀疏表示。为了进行推断和学习，我们开发了一种基于 MC 梯度估计器的摊销变分方法。生成的稀疏表示可通过反向传播进行微分。在多个数据集上针对无监督和监督学习问题进行的实验评估表明了所提方法的有效性。
英文摘要：
Deep latent generative models have attracted increasing attention due to the capacity of combining the strengths of deep learning and probabilistic models in an elegant way. The data representations learned with the models are often continuous and dense. However in many applications, sparse representations are expected, such as learning sparse high dimensional embedding of data in an unsupervised setting, and learning multi-labels from thousands of candidate tags in a supervised setting. In some scenarios, there could be further restriction on degree of sparsity: the number of non-zero features of a representation cannot be larger than a pre-defined threshold $L_0$. In this paper we propose a sparse deep latent generative model SDLGM to explicitly model degree of sparsity and thus enable to learn the sparse structure of the data with the quantified sparsity constraint. The resulting sparsity of a representation is not fixed, but fits to the observation itself under the pre-defined restriction. In particular, we introduce to each observation $i$ an auxiliary random variable $L_i$, which models the sparsity of its representation. The sparse representations are then generated with a two-step sampling process via two Gumbel-Softmax distributions. For inference and learning, we develop an amortized variational method based on MC gradient estimator. The resulting sparse representations are differentiable with backpropagation. The experimental evaluation on multiple datasets for unsupervised and supervised learning problems shows the benefits of the proposed method.
Beijing University of Posts and Telecommunications（北京邮电大学）
Cross-domain recommendation has been proposed to transfer user behavior pattern by pooling together the rating data from multiple domains to alleviate the sparsity problem appearing in single rating domains. However, previous models only assume that multiple domains share a latent common rating pattern based on the user-item co-clustering. To capture diversities among different domains, we propose a novel Probabilistic Cluster-level Latent Factor (PCLF) model to improve the cross-domain recommendation performance. Experiments on several real world datasets demonstrate that our proposed model o…
We address the problem of compressed sensing using a deep generative prior model and consider both linear and learned nonlinear sensing mechanisms, where the nonlinear one involves either a fully connected neural network or a convolutional neural network. Recently, it has been argued that the distribution of natural images do not lie in a single manifold but rather lie in a union of several submanifolds. We propose a sparsity-driven latent space sampling (SDLSS) framework and develop a proximal meta-learning (PML) algorithm to enforce sparsity in the latent space. SDLSS allows the range-space …
Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；LORIA（洛林计算机科学及其应用实验室）
Structuring the latent space in probabilistic deep generative models, e.g., variational autoencoders (VAEs), is important to yield more expressive models and interpretable representations, and to avoid overfitting. One way to achieve this objective is to impose a sparsity constraint on the latent variables, e.g., via a Laplace prior. However, such approaches usually complicate the training phase, and they sacrifice the reconstruction quality to promote sparsity. In this paper, we propose a simple yet effective methodology to structure the latent space via a sparsity-promoting dictionary model,…
Instituto de Telecomunicações（电信研究所）；University of Amsterdam（阿姆斯特丹大学）；Instituto Superior Técnico（高等技术学院）；Unbabel（Unbabel公司）
Training neural network models with discrete (categorical or structured) latent variables can be computationally challenging, due to the need for marginalization over large or combinatorial sets. To circumvent this issue, one typically resorts to sampling-based approximations of the true marginal, requiring noisy gradient estimators (e.g., score function estimator) or continuous relaxations with lower-variance reparameterized gradients (e.g., Gumbel-Softmax). In this paper, we propose a new training strategy which replaces these estimators by an exact yet efficient marginalization. To achieve …
German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））；Goethe University Frankfurt（法兰克福歌德大学）；Frankfurt Cancer Institute（法兰克福癌症研究所）
Many real-world systems are described not only by data from a single source but via multiple data views. In genomic medicine, for instance, patients can be characterized by data from different molecular layers. Latent variable models with structured sparsity are a commonly used tool for disentangling variation within and across data views. However, their interpretability is cumbersome since it requires a direct inspection and interpretation of each factor from domain experts. Here, we propose MuVI, a novel multi-view latent variable model based on a modified horseshoe prior for modeling struct…
Mila（米拉人工智能研究所）；DIRO, Université de Montréal（蒙特利尔大学计算机科学与运筹学系）；ServiceNow Research（ServiceNow研究院）；Tübingen AI Center, University of Tübingen（蒂宾根大学蒂宾根人工智能中心）；Google Research（谷歌研究院）
This work introduces a novel principle we call disentanglement via mechanism sparsity regularization, which can be applied when the latent factors of interest depend sparsely on past latent factors and/or observed auxiliary variables. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors and the sparse causal graphical model that relates them. We develop a rigorous identifiability theory, building on recent nonlinear independent component analysis (ICA) results, that formalizes this principle and shows how the latent variables ca…
Michigan State University（密歇根州立大学）；Argonne National Laboratory（阿贡国家实验室）；Google Research（谷歌研究院）
The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a prior distribution that fixes the dimensionality across all the data can restrict the flexibility of this approach for learning robust representations. We present a novel sparsity-inducing spike-slab categorical prior that uses sparsity as a mechanism to provide the flexibility that allows each data point to learn its own dimension distribution. In addition, it …
Learning Discrete Latent Variable Structures with Tensor Rank Conditions
School of Computer Science, Guangdong University of Technology（广东工业大学计算机学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Beijing Technology and Business University（北京工商大学）；Zhejiang University（浙江大学）；Carnegie Mellon University（卡内基梅隆大学）
Carnegie Mellon University（卡内基梅隆大学）；Indian Institute of Science（印度科学研究所）；Microsoft Research India（微软印度研究院）
We consider the problem of learning a latent $k$-vertex simplex $K\subset\mathbb{R}^d$, given access to $A\in\mathbb{R}^{d\times n}$, which can be viewed as a data matrix with $n$ points that are obtained by randomly perturbing latent points in the simplex $K$ (potentially beyond $K$). A large class of latent variable models, such as adversarial clustering, mixed membership stochastic block models, and topic models can be cast as learning a latent simplex. Bhattacharyya and Kannan (SODA, 2020) give an algorithm for learning such a latent simplex in time roughly $O(k\cdot\textrm{nnz}(A))$, wher…
SPARLING: Learning Latent Representations with Extremely Sparse Activations
Department of Electrical Engineering and Computer Science（电气工程与计算机科学系）；Massachusetts Institute of Technology（麻省理工学院）；Department of Computer and Information Science（计算机与信息科学系）；University of Pennsylvania（宾夕法尼亚大学）
Autoregressive sequence models based on deep neural networks, such as RNNs, Wavenet and the Transformer attain state-of-the-art results on many tasks. However, they are difficult to parallelize and are thus slow at processing long sequences. RNNs lack parallelism both during training and decoding, while architectures like WaveNet and Transformer are much more parallelizable during training, yet still operate sequentially during decoding. Inspired by [arxiv:1711.00937], we present a method to extend sequence models using discrete latent variables that makes decoding much more parallelizable. We…
Conventional methods of estimating latent behaviour generally use attitudinal questions which are subjective and these survey questions may not always be available. We hypothesize that an alternative approach can be used for latent variable estimation through an undirected graphical models. For instance, non-parametric artificial neural networks. In this study, we explore the use of generative non-parametric modelling methods to estimate latent variables from prior choice distribution without the conventional use of measurement indicators. A restricted Boltzmann machine is used to represent la…
Learning multiple tasks across heterogeneous domains is a challenging problem since the feature space may not be the same for different tasks. We assume the data in multiple tasks are generated from a latent common domain via sparse domain transforms and propose a latent probit model (LPM) to jointly learn the domain transforms, and the shared probit classifier in the common domain. To learn meaningful task relatedness and avoid over-fitting in classification, we introduce sparsity in the domain transforms matrices, as well as in the common classifier. We derive theoretical bounds for the esti…
University of Cambridge（剑桥大学）；Monash University（莫纳什大学）；Imperial College London（帝国理工学院）
It has been long known that sparsity is an effective inductive bias for learning efficient representation of data in vectors with fixed dimensionality, and it has been explored in many areas of representation learning. Of particular interest to this work is the investigation of the sparsity within the VAE framework which has been explored a lot in the image domain, but has been lacking even a basic level of exploration in NLP. Additionally, NLP is also lagging behind in terms of learning sparse representations of large units of text e.g., sentences. We use the VAEs that induce sparse latent re…
In many sequence learning tasks, such as program synthesis and document summarization, a key problem is searching over a large space of possible output sequences. We propose to learn representations of the outputs that are specifically meant for search: rich enough to specify the desired output but compact enough to make search more efficient. Discrete latent codes are appealing for this purpose, as they naturally allow sophisticated combinatorial search strategies. The latent codes are learned using a self-supervised learning principle, in which first a discrete autoencoder is trained on the …
Existing techniques for certifying the robustness of models for discrete data either work only for a small class of models or are general at the expense of efficiency or tightness. Moreover, they do not account for sparsity in the input which, as our findings show, is often essential for obtaining non-trivial guarantees. We propose a model-agnostic certificate based on the randomized smoothing framework which subsumes earlier work and is tight, efficient, and sparsity-aware. Its computational complexity does not depend on the number of discrete categories or the dimension of the input (e.g. th…
Aston University（阿斯顿大学）；University of Sussex（萨塞克斯大学）；University of Birmingham（伯明翰大学）；Massachusetts Institute of Technology（麻省理工学院）
Ubiquitous linear Gaussian exploratory tools such as principle component analysis (PCA) and factor analysis (FA) remain widely used as tools for: exploratory analysis, pre-processing, data visualization and related tasks. However, due to their rigid assumptions including crowding of high dimensional data, they have been replaced in many settings by more flexible and still interpretable latent feature models. The Feature allocation is usually modelled using discrete latent variables assumed to follow either parametric Beta-Bernoulli distribution or Bayesian nonparametric prior. In this work we …
A Sparsity Principle for Partially Observable Causal Representation Learning
University of Amsterdam（阿姆斯特丹大学）；Institute of Science and Technology Austria（奥地利科技学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Samsung - SAIT AI Lab, Montreal（三星SAIT AI实验室（蒙特利尔））；Mila, Université de Montréal（蒙特利尔大学Mila研究所）；ServiceNow Research（ServiceNow研究院）；Seminar for Statistics, ETH Zürich（苏黎世联邦理工学院统计研究所）
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；Telekom Innovation Laboratories at BGU（BGU德国电信创新实验室）；Stern School of Business, New York University（纽约大学斯特恩商学院）
Context-aware recommender systems (CARSs) apply sensing and analysis of user context in order to provide personalized services. Adding context to a recommendation model is challenging, since the addition of context may increases both the dimensionality and sparsity of the model. Recent research has shown that modeling contextual information as a latent vector may address the sparsity and dimensionality challenges. We suggest a new latent modeling of sequential context by generating sequences of contextual information and reducing their contextual space to a compressed latent space.We train a l…
Time Series Domain Adaptation via Latent Invariant Causal Mechanism
School of Computer Science, Guangdong University of Technology（广东工业大学计算机学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computing, Guangdong University of Technology（广东工业大学计算机学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究所）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
DeepMind（深度思维）；Microsoft Research New England（微软新英格兰研究院）
Stochastic gradient-based optimisation for discrete latent variable models is challenging due to the high variance of gradients. We introduce a variance reduction technique for score function estimators that makes use of double control variates. These control variates act on top of a main control variate, and try to further reduce the variance of the overall estimator. We develop a double control variate for the REINFORCE leave-one-out estimator using Taylor expansions. For training discrete latent variable models, such as variational autoencoders with binary latent variables, our approach add…
We consider the problem of molecular graph generation using deep models. While graphs are discrete, most existing methods use continuous latent variables, resulting in inaccurate modeling of discrete graph structures. In this work, we propose GraphDF, a novel discrete latent variable model for molecular graph generation based on normalizing flow methods. GraphDF uses invertible modulo shift transforms to map discrete latent variables to graph nodes and edges. We show that the use of discrete latent variables reduces computational costs and eliminates the negative effect of dequantization. Comp…
University at Buffalo（布法罗大学）；Criteo（Criteo公司）；Duke University（杜克大学）
Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. However, the sequential text generation common pitfall with VAEs is that the model tends to ignore latent variables with a strong auto-regressive decoder. In this paper, we propose a principled approach to alleviate this issue by applying a discretized bottleneck to enforce an implicit latent feature matching in a more compact latent space. We impose a shared discrete latent space where each input is learned to choose a combination of latent atoms as a regularized latent representation. Our model endows …
Inria（法国国家信息与自动化技术研究院）；PSL Research University（巴黎社会科学高等研究院）；CMAP（巴黎高等理工学院应用数学与计算科学实验室）；Ecole Polytechnique Palaiseau, France（法国巴黎高等理工学院Palaiseau分校）；Sakana AI Tokyo, Japan（日本东京Sakana AI公司）
Generative Modeling of Discrete Data Using Geometric Latent Subspaces
Institute for Mathematics, Heidelberg University（海德堡大学数学研究所）；Mannheim Institute for Intelligent Systems in Medicine, Heidelberg University（海德堡大学曼海姆智能医学系统研究所）；IWR, Heidelberg University（海德堡大学IWR研究所）；Research Station Geometry and Dynamics, Heidelberg University（海德堡大学几何与动力学研究站）
Discrete latent variables are considered important for real world data, which has motivated research on Variational Autoencoders (VAEs) with discrete latents. However, standard VAE training is not possible in this case, which has motivated different strategies to manipulate discrete distributions in order to train discrete VAEs similarly to conventional ones. Here we ask if it is also possible to keep the discrete nature of the latents fully intact by applying a direct discrete optimization for the encoding model. The approach is consequently strongly diverting from standard VAE-training by si…
Normalising flows (NFs) for discrete data are challenging because parameterising bijective transformations of discrete variables requires predicting discrete/integer parameters. Having a neural network architecture predict discrete parameters takes a non-differentiable activation function (eg, the step function) which precludes gradient-based learning. To circumvent this non-differentiability, previous work has employed biased proxy gradients, such as the straight-through estimator. We present an unbiased alternative where rather than deterministically parameterising one transformation, we pre…
Harvard University（哈佛大学）；University of Texas at Austin（得克萨斯大学奥斯汀分校）
Originally designed to model text, topic modeling has become a powerful tool for uncovering latent structure in domains including medicine, finance, and vision. The goals for the model vary depending on the application: in some cases, the discovered topics may be used for prediction or some other downstream task. In other cases, the content of the topic itself may be of intrinsic scientific interest. Unfortunately, even using modern sparse techniques, the discovered topics are often difficult to interpret due to the high dimensionality of the underlying space. To improve topic interpretability…
Many important problems in the real world don't have unique solutions. It is thus important for machine learning models to be capable of proposing different plausible solutions with meaningful probability measures. In this work we introduce such a deep learning framework that learns the one-to-many mappings between the inputs and outputs, together with faithful uncertainty measures. We call our framework {\it modal uncertainty estimation} since we model the one-to-many mappings to be generated through a set of discrete latent variables, each representing a latent mode hypothesis that explains …
University of Hyderabad（海得拉巴大学）；University of Delhi（德里大学）；Mahindra University（马恒达大学）
Recommender systems based on collaborative filtering play a vital role in many E-commerce applications as they guide the user in finding their items of interest based on the user's past transactions and feedback of other similar customers. Data Sparsity is one of the major drawbacks with collaborative filtering technique arising due to the less number of transactions and feedback data. In order to reduce the sparsity problem, techniques called transfer learning/cross-domain recommendation has emerged. In transfer learning methods, the data from other dense domain(s) (source) is considered in o…
Accelerating Matrix Factorization by Dynamic Pruning for Fast Recommendation
School of Computer Engineering and Science, Shanghai University（上海大学计算机工程与科学学院）；College of Integrated Circuits and Optoelectronic Chips, School of Computer Engineering and Science, Shenzhen Technology University（深圳科技大学集成电路与光电芯片学院、计算机工程与科学学院）
Korea University（高丽大学）；Queen Mary University of London（伦敦玛丽女王大学）；Korea National Open University（韩国国立开放大学）
In recent years, neural network based methods have been proposed as a method that cangenerate representations from music, but they are not human readable and hardly analyzable oreditable by a human. To address this issue, we propose a novel method to learn source-awarelatent representations of music through Vector-Quantized Variational Auto-Encoder(VQ-VAE).We train our VQ-VAE to encode an input mixture into a tensor of integers in a discrete latentspace, and design them to have a decomposed structure which allows humans to manipulatethe latent vector in a source-aware manner. This paper also s…
Deep latent variable models are powerful tools for representation learning. In this paper, we adopt the deep information bottleneck model, identify its shortcomings and propose a model that circumvents them. To this end, we apply a copula transformation which, by restoring the invariance properties of the information bottleneck method, leads to disentanglement of the features in the latent space. Building on that, we show how this transformation translates to sparsity of the latent space in the new model. We evaluate our method on artificial and real data.
University College London（伦敦大学学院）；ASI Data Science（ASI数据科学）；Alan Turing Institute（艾伦·图灵研究所）
Generative models with both discrete and continuous latent variables are highly motivated by the structure of many real-world data sets. They present, however, subtleties in training often manifesting in the discrete latent being under leveraged. In this paper, we show that such models are more amenable to training when using the Optimal Transport framework of Wasserstein Autoencoders. We find our discrete latent variable to be fully leveraged by the model when trained, without any modifications to the objective function or significant fine tuning. Our model generates comparable samples to oth…
University of Amsterdam（阿姆斯特丹大学）；INSA Rennes（雷恩国立应用科学学院）；IRISA（法国信息与随机系统研究所）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Université de Rennes（雷恩大学）；Amazon（亚马逊）；Instituto Superior Técnico（高等技术学院）；Universidade de Lisboa（里斯本大学）；Instituto de Telecomunicações（电信研究所）；Unbabel
Google Brain（谷歌大脑）；DeepMind；University of Oxford（牛津大学）
Learning in models with discrete latent variables is challenging due to high variance gradient estimators. Generally, approaches have relied on control variates to reduce the variance of the REINFORCE estimator. Recent work (Jang et al. 2016, Maddison et al. 2016) has taken a different approach, introducing a continuous relaxation of discrete variables to produce low-variance, but biased, gradient estimates. In this work, we combine the two approaches through a novel control variate that produces low-variance, \emph{unbiased} gradient estimates. Then, we introduce a modification to the continu…
