Harmonized Multimodal Learning with Gaussian Process Latent Variable Models
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Noah’s Ark Lab（诺亚方舟实验室）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Multimodal learning aims to discover the relationship between multiple modalities. It has become an important research topic due to extensive multimodal applications such as cross-modal retrieval. This paper attempts to address the modality heterogeneity problem based on Gaussian process latent variable models (GPLVMs) to represent multimodal data in a common space. Previous multimodal GPLVM extensions generally adopt individual learning schemes on latent representations and kernel hyperparameters, which ignore their intrinsic relationship. To exploit strong complementarity among different modalities and GPLVM components, we develop a novel learning scheme called Harmonization, where latent model parameters are jointly learned from each other. Beyond the correlation fitting or intra-modal structure preservation paradigms widely used in existing studies, the harmonization is derived in a model-driven manner to encourage the agreement between modality-specific GP kernels and the similarity of latent representations. We present a range of multimodal learning models by incorporating the harmonization mechanism into several representative GPLVM-based approaches. Experimental results on four benchmark datasets show that the proposed models outperform the strong baselines for cross-modal retrieval tasks, and that the harmonized multimodal learning method is superior in discovering semantically consistent latent representation.
Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning
Responsible AI Research Centre（负责任人工智能研究中心）；Australian Institute for Machine Learning, Adelaide University（阿德莱德大学澳大利亚机器学习研究所）；School of Computer Science and Engineering, The University of New South Wales（新南威尔士大学计算机科学与工程学院）；School of Mathematics and Statistics, The University of Melbourne（墨尔本大学数学与统计学院）；Department of Philosophy, Carnegie Mellon University（卡内基梅隆大学哲学系）
University of Electronic Science and Technology of China（电子科技大学）
We study the problem of multimodal generative modelling of images based on generative adversarial networks (GANs). Despite the success of existing methods, they often ignore the underlying structure of vision data or its multimodal generation characteristics. To address this problem, we introduce the Dirichlet prior for multimodal image generation, which leads to a new Latent Dirichlet Allocation based GAN (LDAGAN). In detail, for the generative process modelling, LDAGAN defines a generative mode for each sample, determining which generative sub-process it belongs to. For the adversarial train…
Discrete latent spaces in variational autoencoders have been shown to effectively capture the data distribution for many real-world problems such as natural language understanding, human intent prediction, and visual scene representation. However, discrete latent spaces need to be sufficiently large to capture the complexities of real-world data, rendering downstream tasks computationally challenging. For instance, performing motion planning in a high-dimensional latent representation of the environment could be intractable. We consider the problem of sparsifying the discrete latent space of a…
Multi-modal generative models represent an important family of deep models, whose goal is to facilitate representation learning on data with multiple views or modalities. However, current deep multi-modal models focus on the inference of shared representations, while neglecting the important private aspects of data within individual modalities. In this paper, we introduce a disentangled multi-modal variational autoencoder (DMVAE) that utilizes disentangled VAE strategy to separate the private and shared latent spaces of multiple modalities. We specifically consider the instance where the laten…
One of the major shortcomings of variational autoencoders is the inability to produce generations from the individual modalities of data originating from mixture distributions. This is primarily due to the use of a simple isotropic Gaussian as the prior for the latent code in the ancestral sampling procedure for the data generations. We propose a novel formulation of variational autoencoders, conditional prior VAE (CP-VAE), which learns to differentiate between the individual mixture components and therefore allows for generations from the distributional data clusters. We assume a two-level ge…
Multimodal image-to-image translation (I2IT) aims to learn a conditional distribution that explores multiple possible images in the target domain given an input image in the source domain. Conditional generative adversarial networks (cGANs) are often adopted for modeling such a conditional distribution. However, cGANs are prone to ignore the latent code and learn a unimodal distribution in conditional image synthesis, which is also known as the mode collapse issue of GANs. To solve the problem, we propose a simple yet effective method that explicitly estimates and maximizes the mutual informat…
Multimodal learning has been lacking principled ways of combining information from different modalities and learning a low-dimensional manifold of meaningful representations. We study multimodal learning and sensor fusion from a latent variable perspective. We first present a regularized recurrent attention filter for sensor fusion. This algorithm can dynamically combine information from different types of sensors in a sequential decision making task. Each sensor is bonded with a modular neural network to maximize utility of its own information. A gating modular neural network dynamically gene…
Georgia Institute of Technology（佐治亚理工学院）；Arizona State University（亚利桑那州立大学）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）；MS Technologies Corporation（MS科技公司）
A tutorial on discovering and quantifying the effect of latent causal sources of multimodal EHR data
Department of Biomedical Informatics Vanderbilt University Medical Center（生物医学信息学系范德堡大学医学中心）；Department of Biomedical Informatics University of Pittsburgh（生物医学信息学系匹兹堡大学）；Departments of Medicine & Biomedical Informatics Vanderbilt University Medical Center（医学与生物医学信息学系范德堡大学医学中心）；Departments of Biomedical Informatics & Computer Science Vanderbilt University Medical Center & Vanderbilt University（生物医学信息学与计算机科学系范德堡大学医学中心及范德堡大学）
Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）；Leibniz Institute for Prevention Research and Epidemiology – BIPS（莱比锡预防研究与流行病学研究所 – BIPS）；Faculty of Mathematics and Computer Science, University of Bremen（不莱梅大学数学与计算机科学学院）；Faculty of Physics and Electrical Engineering, University of Bremen（不莱梅大学物理与电气工程学院）
IIT Jodhpur（印度理工学院焦特布尔分校）；Flipkart Internet Pvt Ltd（Flipkart互联网私人有限公司）；IIT Delhi（印度理工学院德里分校）
Generative adversarial networks (GANs) have shown remarkable success in generation of data from natural data manifolds such as images. In several scenarios, it is desirable that generated data is well-clustered, especially when there is severe class imbalance. In this paper, we focus on the problem of clustering in generated space of GANs and uncover its relationship with the characteristics of the latent space. We derive from first principles, the necessary and sufficient conditions needed to achieve faithful clustering in the GAN framework: (i) presence of a multimodal latent space with adju…
Analyzing Multimodal Integration in the Variational Autoencoder from an Information-Theoretic Perspective
Hamburg University of Technology（汉堡工业大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Federal Institute for Materials Research and Testing（联邦材料研究与测试研究所）；Santa Fe Institute（圣塔菲研究所）；Leipzig University（莱比锡大学）
Hong Kong University of Science and Technology（香港科技大学）；Clova AI Research, NAVER（NAVER Clova AI研究）；CIFAR Azrieli Global Scholar（CIFAR Azrieli全球学者）
Variational autoencoders~(VAEs) have shown a promise in data-driven conversation modeling. However, most VAE conversation models match the approximate posterior distribution over the latent variables to a simple prior such as standard normal distribution, thereby restricting the generated responses to a relatively simple (e.g., unimodal) scope. In this paper, we propose DialogWAE, a conditional Wasserstein autoencoder~(WAE) specially designed for dialogue modeling. Unlike VAEs that impose a simple distribution over the latent variables, DialogWAE models the distribution of data by training a G…
Latent Reconstruction from Generated Data for Multimodal Misinformation Detection
Information Technology Institute, Centre for Research & Technology, Hellas（信息科技研究所，研究中心，希腊）；Department of Electrical & Computer Engineering, Aristotle University of Thessaloniki（电气与计算机工程系，亚里士多德大学）
Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation a…
University of Toronto Institute for Aerospace Studies (UTIAS)（多伦多大学航空航天研究所）
Sequential modelling of high-dimensional data is an important problem that appears in many domains including model-based reinforcement learning and dynamics identification for control. Latent variable models applied to sequential data (i.e., latent dynamics models) have been shown to be a particularly effective probabilistic approach to solve this problem, especially when dealing with images. However, in many application areas (e.g., robotics), information from multiple sensing modalities is available -- existing latent dynamics methods have not yet been extended to effectively make use of suc…
LatentVerse: A Framework for Understanding Shared and Modality-Specific Information in Multimodal Latent Representations
Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学布罗德研究所）；Massachusetts Institute of Technology（麻省理工学院）；The Schmidt Center, Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学布罗德研究所施密特中心）
Multimodal data arise in various applications where information about the same phenomenon is acquired from multiple sensors and across different imaging modalities. Learning from multimodal data is of great interest in machine learning and statistics research as this offers the possibility of capturing complementary information among modalities. Multimodal modeling helps to explain the interdependence between heterogeneous data sources, discovers new insights that may not be available from a single modality, and improves decision-making. Recently, coupled matrix-tensor factorization has been i…
CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning
School of Electronics and Information Technology, Sun Yat-Sen University（中山大学电子与信息学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Computer Science, South China Normal University（华南师范大学计算机科学学院）；Desay SV Automotive Co., Ltd（德赛股份有限公司）；Pazhou Laboratory（琶洲实验室）
