Learning disentangled and interpretable representations is an important step towards accomplishing comprehensive data representations on the manifold. In this paper, we propose a novel representation learning algorithm which combines the inference abilities of Variational Autoencoders (VAE) with the generalization capability of Generative Adversarial Networks (GAN). The proposed model, called InfoVAEGAN, consists of three networks~: Encoder, Generator and Discriminator. InfoVAEGAN aims to jointly learn discrete and continuous interpretable representations in an unsupervised manner by using two different data-free log-likelihood functions onto the variables sampled from the generator's distribution. We propose a two-stage algorithm for optimizing the inference network separately from the generator training. Moreover, we enforce the learning of interpretable representations through the maximization of the mutual information between the existing latent variables and those created through generative and inference processes.
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Learning interpretable and disentangled representations of data is a key topic in machine learning research. Variational Autoencoder (VAE) is a scalable method for learning directed latent variable models of complex data. It employs a clear and interpretable objective that can be easily optimized. However, this objective does not provide an explicit measure for the quality of latent variable representations which may result in their poor quality. We propose Variational Mutual Information Maximization Framework for VAE to address this issue. In comparison to other methods, it provides an explic…
Are deep convolutional neural networks (CNNs) for image classification explainable by utility maximization with information acquisition costs? We demonstrate that deep CNNs behave equivalently (in terms of necessary and sufficient conditions) to rationally inattentive utility maximizers, a generative model used extensively in economics for human decision making. Our claim is based by extensive experiments on 200 deep CNNs from 5 popular architectures. The parameters of our interpretable model are computed efficiently via convex feasibility algorithms. As an application, we show that our econom…
For natural frequency optimization of engineering structures, cellular composites have been shown to possess an edge over solid. However, existing multiscale design methods for cellular composites are either computationally exhaustive or confined to a single class of microstructures. In this paper, we propose a data-driven topology optimization (TO) approach to enable the multiscale design of cellular structures with various choices of microstructure classes. The key component is a newly proposed latent-variable Gaussian process (LVGP) model through which different classes of microstructures a…
Time-series representation learning is a fundamental task for time-series analysis. While significant progress has been made to achieve accurate representations for downstream applications, the learned representations often lack interpretability and do not expose semantic meanings. Different from previous efforts on the entangled feature space, we aim to extract the semantic-rich temporal correlations in the latent interpretable factorized representation of the data. Motivated by the success of disentangled representation learning in computer vision, we study the possibility of learning semant…
School of Astronautics, Beihang University（北京航空航天大学宇航学院）
Disentanglement learning aims to construct independent and interpretable latent variables in which generative models are a popular strategy. InfoGAN is a classic method via maximizing Mutual Information (MI) to obtain interpretable latent variables mapped to the target space. However, it did not emphasize independent characteristic. To explicitly infer latent variables with inter-independence, we propose a novel GAN-based disentanglement framework via embedding Orthogonal Basis Expansion (OBE) into InfoGAN network (Inference-InfoGAN) in an unsupervised way. Under the OBE module, one set of ort…
Institute of Informatics and Telecommunications（信息与通信研究所）
Learning disentangled representations without supervision or inductive biases, often leads to non-interpretable or undesirable representations. On the other hand, strict supervision requires detailed knowledge of the true generative factors, which is not always possible. In this paper, we consider weak supervision by means of high-level labels that are not assumed to be explicitly related to the ground truth factors. Such labels, while being easier to acquire, can also be used as inductive biases for algorithms to learn more interpretable or alternative disentangled representations. To this en…
Probabilistic graphical modeling (PGM) provides a framework for formulating an interpretable generative process of data and expressing uncertainty about unknowns, but it lacks flexibility. Deep learning (DL) is an alternative framework for learning from data that has achieved great empirical success in recent years. DL offers great flexibility, but it lacks the interpretability and calibration of PGM. This thesis develops deep probabilistic graphical modeling (DPGM.) DPGM consists in leveraging DL to make PGM more flexible. DPGM brings about new methods for learning from data that exhibit the …
Learning disentangled representations leads to interpretable models and facilitates data generation with style transfer, which has been extensively studied on static data such as images in an unsupervised learning framework. However, only a few works have explored unsupervised disentangled sequential representation learning due to challenges of generating sequential data. In this paper, we propose recurrent Wasserstein Autoencoder (R-WAE), a new framework for generative modeling of sequential data. R-WAE disentangles the representation of an input sequence into static and dynamic factors (i.e.…
Switching dynamical systems provide a powerful, interpretable modeling framework for inference in time-series data in, e.g., the natural sciences or engineering applications. Since many areas, such as biology or discrete-event systems, are naturally described in continuous time, we present a model based on an Markov jump process modulating a subordinated diffusion process. We provide the exact evolution equations for the prior and posterior marginal densities, the direct solutions of which are however computationally intractable. Therefore, we develop a new continuous-time variational inferenc…
Pairwise models like the Ising model or the generalized Potts model have found many successful applications in fields like physics, biology, and economics. Closely connected is the problem of inverse statistical mechanics, where the goal is to infer the parameters of such models given observed data. An open problem in this field is the question of how to train these models in the case where the data contain additional higher-order interactions that are not present in the pairwise model. In this work, we propose an approach based on Energy-Based Models and pseudolikelihood maximization to addre…
Physically Interpretable Representation Learning with Gaussian Mixture Variational AutoEncoder (GM-VAE)
Institute for Computational and Mathematical Engineering, Stanford University（计算与数学工程研究所，斯坦福大学）；Department of Mechanical Engineering, Stanford University（机械工程系，斯坦福大学）；National Renewable Energy Laboratory（国家可再生能源实验室）；Ann and H.J. Smead Department of Aerospace Engineering Sciences, University of Colorado, Boulder（安与H.J.斯迈德航空航天工程科学系，科罗拉多大学，博尔德）
Enhancing Topic Interpretability for Neural Topic Modeling through Topic-wise Contrastive Learning
Peking University（北京大学）；Software Engineering Institution（软件工程研究院）；Key Lab of High Confidence Software Technologies, Ministry of Education（高可信软件技术重点实验室（教育部））；Center on Frontiers of Computing Studies（计算机前沿研究中心）；National Engineering Research Center of Software Engineering（国家软件工程工程研究中心）
Learning Fingerprints for Medical Time Series with Redundancy-Constrained Information Maximization
University of Arizona, AZ, USA（亚利桑那大学）；Microsoft Research, Shanghai, China（微软研究院）；Clemson University, SC, USA（克莱姆森大学）；Villanova University, IL, USA（维拉诺瓦大学）
Interpretable and explainable machine learning has seen a recent surge of interest. We focus on safety as a key motivation behind the surge and make the relationship between interpretability and safety more quantitative. Toward assessing safety, we introduce the concept of maximum deviation via an optimization problem to find the largest deviation of a supervised learning model from a reference model regarded as safe. We then show how interpretability facilitates this safety assessment. For models including decision trees, generalized linear and additive models, the maximum deviation can be co…
Graph representation learning based on graph neural networks (GNNs) can greatly improve the performance of downstream tasks, such as node and graph classification. However, the general GNN models do not aggregate node information in a hierarchical manner, and can miss key higher-order structural features of many graphs. The hierarchical aggregation also enables the graph representations to be explainable. In addition, supervised graph representation learning requires labeled data, which is expensive and error-prone. To address these issues, we present an unsupervised graph representation learn…
Enhancing Pre-trained Representation Classifiability can Boost its Interpretability
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS（智能信息处理重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Institute of Technology, Weihai（哈尔滨工业大学威海学院）；Peng Cheng Laboratory（鹏城实验室）；Huawei Inc.（华为公司）
Normative Networks for Source Separation via Local Plasticity and Dendritic Computation
Gatsby Computational Neuroscience Unit, University College London, UK（Gatsby计算神经科学单元，伦敦大学学院，英国）；Department of Computer Science, University of Oxford, UK（计算机科学系，牛津大学，英国）；Brain Network Dynamics Unit, University of Oxford, UK（脑网络动力学单元，牛津大学，英国）；MRC Centre of Research Excellence in Restorative Neural Dynamics, UK（英国医学研究委员会修复神经动力学研究卓越中心）
We propose a novel learning framework based on neural mean-field dynamics for inference and estimation problems of diffusion on networks. Our new framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities, which renders a delay differential equation with memory integral approximated by learnable time convolution operators, resulting in a highly structured and interpretable RNN. Directly using cascade data, our framework can jointly learn the structure of the diffusion network and the evolution of infection probabilities, which are corn…
Tsinghua University（清华大学）；University of Technology Sydney（悉尼科技大学）
Hawkes process provides an effective statistical framework for analyzing the time-dependent interaction of neuronal spiking activities. Although utilized in many real applications, the classic Hawkes process is incapable of modelling inhibitory interactions among neurons. Instead, the nonlinear Hawkes process allows for a more flexible influence pattern with excitatory or inhibitory interactions. In this paper, three sets of auxiliary latent variables (Pólya-Gamma variables, latent marked Poisson processes and sparsity variables) are augmented to make functional connection weights in a Gaussia…
Identifiable Bayesian Deep Generative Copulas with Unknown Layer Widths for Data with Arbitrary Marginal Distributions
Department of Statistics and Data Science（统计与数据科学系）；Washington University in St. Louis（圣路易斯华盛顿大学）；Department of Statistics（统计系）；Columbia University（哥伦比亚大学）
We propose a generative model for single-channel EEG that incorporates the constraints experts actively enforce during visual scoring. The framework takes the form of a dynamic Bayesian network with depth in both the latent variables and the observation likelihoods-while the hidden variables control the durations, state transitions, and robustness, the observation architectures parameterize Normal-Gamma distributions. The resulting model allows for time series segmentation into local, reoccurring dynamical regimes by exploiting probabilistic models and deep learning. Unlike typical detectors, …
Massachusetts Institute of Technology（麻省理工学院）；Optum Labs（Optum实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Sum-product networks (SPNs) have recently emerged as a novel deep learning architecture enabling highly efficient probabilistic inference. Since their introduction, SPNs have been applied to a wide range of data modalities and extended to time-sequence data. In this paper, we propose a general framework for modelling sequential treatment decision-making behaviour and treatment response using recurrent sum-product networks (RSPNs). Models developed using our framework benefit from the full range of RSPN capabilities, including the abilities to model the full distribution of the data, to seamles…
Central Institute of Mental Health（中央精神卫生研究所）；Heidelberg University（海德堡大学）；Technical University Berlin（柏林工业大学）
Empirically observed time series in physics, biology, or medicine, are commonly generated by some underlying dynamical system (DS) which is the target of scientific interest. There is an increasing interest to harvest machine learning methods to reconstruct this latent DS in a data-driven, unsupervised way. In many areas of science it is common to sample time series observations from many data modalities simultaneously, e.g. electrophysiological and behavioral time series in a typical neuroscience experiment. However, current machine learning tools for reconstructing DSs usually focus on just …
GEM-T: Generative Tabular Data via Fitting Moments
Department of Pathology at Beth Israel Deaconess Medical Center (BIDMC)（波士顿贝塞斯达德aconess医疗中心病理学系）；Department of Medicine, the Bakar Computational Health Sciences Institute, and the UCSF UC Berkeley Joint Program for Computational Precision Health at the University of California San Francisco（旧金山大学医学系、Bakar计算健康科学研究所及UCSF-伯克利联合计算精准健康项目）；Department of Pathology and the Division of Clinical Informatics, Department of Medicine, BIDMC and with Harvard Medical School（波士顿贝塞斯达德aconess医疗中心病理学系及医学系临床信息学分会、哈佛医学院）
GEM-T 基于最大熵原理直接拟合表格数据列间 n 阶交互，以极少的可训练参数在 34 个数据集中 68% 上达到或超越深度神经网络方法，为结构化数据提供轻量高性能生成新方向。
When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Huawei Technologies Co., Ltd.（华为技术有限公司）；City University of Hong Kong（香港城市大学）；School of Cyber Science and Technology, Sun Yat-sen University（中山大学信息科学与技术学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）
