Multimodal learning with deep Boltzmann machines (DBMs) is an generative approach to fuse multimodal inputs, and can learn the shared representation via Contrastive Divergence (CD) for classification and information retrieval tasks. However, it is a 2-fan DBM model, and cannot effectively handle multiple prediction tasks. Moreover, this model cannot recover the hidden representations well by sampling from the conditional distribution when more than one modalities are missing. In this paper, we propose a K-fan deep structure model, which can handle the multi-input and muti-output learning problems effectively. In particular, the deep structure has K-branch for different inputs where each branch can be composed of a multi-layer deep model, and a shared representation is learned in an discriminative manner to tackle multimodal tasks. Given the deep structure, we propose two objective functions to handle two multi-input and multi-output tasks: joint visual restoration and labeling, and the multi-view multi-calss object recognition tasks. To estimate the model parameters, we initialize the deep model parameters with CD to maximize the joint distribution, and then we use backpropagation to update the model according to specific objective function. The experimental results demonstrate that the model can effectively leverages multi-source information and predict multiple tasks well over competitive baselines.
Sandia National Laboratories（桑迪亚国家实验室）；Arizona State University（亚利桑那州立大学）
We introduce physics-informed multimodal autoencoders (PIMA) - a variational inference framework for discovering shared information in multimodal scientific datasets representative of high-throughput testing. Individual modalities are embedded into a shared latent space and fused through a product of experts formulation, enabling a Gaussian mixture prior to identify shared features. Sampling from clusters allows cross-modal generative modeling, with a mixture of expert decoder imposing inductive biases encoding prior scientific knowledge and imparting structured disentanglement of the latent s…
Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, a line of recent work reveals that contrastive learning can invert the data generating process and recover ground truth latent factors shared between views. In this work, we present new identifiability results for multimodal contrastive learning, showing that it is possible to recover shared factors in a more general setup than the multi-view setting studied previously. Specificall…
Multi-modal generative models represent an important family of deep models, whose goal is to facilitate representation learning on data with multiple views or modalities. However, current deep multi-modal models focus on the inference of shared representations, while neglecting the important private aspects of data within individual modalities. In this paper, we introduce a disentangled multi-modal variational autoencoder (DMVAE) that utilizes disentangled VAE strategy to separate the private and shared latent spaces of multiple modalities. We specifically consider the instance where the laten…
LatentVerse: A Framework for Understanding Shared and Modality-Specific Information in Multimodal Latent Representations
Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学布罗德研究所）；Massachusetts Institute of Technology（麻省理工学院）；The Schmidt Center, Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学布罗德研究所施密特中心）
LunarFM: A Shared Multimodal Representation of the Moon's Surface
University of Cambridge（剑桥大学）；German Aerospace Center (DLR)（德国航空航天中心）；SPAIDER SPACE（斯派德太空公司）；Mines Paris - PSL University（巴黎矿业学院 - 巴黎文理研究大学）；University of Bern（伯尔尼大学）；Imperial College London（伦敦帝国理工学院）；European Space Resources Innovation Center (ESRIC)（欧洲空间资源创新中心）；Universidad de Antioquia（安蒂奥基亚大学）
Center for Brain-like Computing and Machine Intelligence（类脑计算与机器智能中心）；Department of Computer Science and Engineering（计算机科学与工程系）；Shanghai Jiao Tong University（上海交通大学）
To enhance the performance of affective models and reduce the cost of acquiring physiological signals for real-world applications, we adopt multimodal deep learning approach to construct affective models from multiple physiological signals. For unimodal enhancement task, we indicate that the best recognition accuracy of 82.11% on SEED dataset is achieved with shared representations generated by Deep AutoEncoder (DAE) model. For multimodal facilitation tasks, we demonstrate that the Bimodal Deep AutoEncoder (BDAE) achieves the mean accuracies of 91.01% and 83.25% on SEED and DEAP datasets, resp…
Multimodal learning is a framework for building models that make predictions based on different types of modalities. Important challenges in multimodal learning are the inference of shared representations from arbitrary modalities and cross-modal generation via these representations; however, achieving this requires taking the heterogeneous nature of multimodal data into account. In recent years, deep generative models, i.e., generative models in which distributions are parameterized by deep neural networks, have attracted much attention, especially variational autoencoders, which are suitable…
MIT Lincoln Laboratory（麻省理工学院林肯实验室）；Harvard University（哈佛大学）
Unsupervised feature learning methods have proven effective for classification tasks based on a single modality. We present multimodal sparse coding for learning feature representations shared across multiple modalities. The shared representations are applied to multimedia event detection (MED) and evaluated in comparison to unimodal counterparts, as well as other feature learning methods such as GMM supervectors and sparse RBM. We report the cross-validated classification accuracy and mean average precision of the MED system trained on features learned from our unimodal and multimodal setting…
John A. Paulson School of Engineering and Applied Sciences（约翰·A·保尔森工程与应用科学学院）；Harvard University（哈佛大学）
Unsupervised methods have proven effective for discriminative tasks in a single-modality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared sparse representation. In particular, we propose the use of joint dictionary learning technique for sparse coding and formulate the joint representation for concision, cross-modal representations (in case of a missing modality), and union of the cross-modal representations. Given…
DRIM: Learning Disentangled Representations from Incomplete Multimodal Healthcare Data
IUCT-Oncopole Institut Claudius Regaud（克劳迪乌斯·雷戈德研究所肿瘤学中心）；IRT Saint Exupéry（圣埃克苏佩里技术研究院）；INSERM UMR 1037, Cancer Research Center of Toulouse (CRCT)（法国国家健康与医学研究院UMR 1037，图卢兹癌症研究中心）
Southwest Jiaotong University（西南交通大学）；Microsoft Research Asia（微软亚洲研究院）；Southwestern University of Finance and Economics（西南财经大学）；Ocean University of China（中国海洋大学）；Tencent（腾讯）
Fusion technique is a key research topic in multimodal sentiment analysis. The recent attention-based fusion demonstrates advances over simple operation-based fusion. However, these fusion works adopt single-scale, i.e., token-level or utterance-level, unimodal representation. Such single-scale fusion is suboptimal because that different modality should be aligned with different granularities. This paper proposes a fusion model named ScaleVLAD to gather multi-Scale representation from text, video, and audio with shared Vectors of Locally Aggregated Descriptors to improve unaligned multimodal s…
MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation
Alibaba Group（阿里巴巴集团）；Peking University（北京大学）；Beijing University of Aeronautics and Astronautics（北京航空航天大学）；Wuhan University, School of Computer Science（武汉大学计算机学院）
EviDep: Uncertainty-Aware Multimodal Depression Estimation via Disentangled Evidential Learning
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）；Department of Psychiatry, The First Affiliated Hospital of University of Science and Technology of China, Division of Life Sciences and Medicine, University of Science and Technology of China（中国科学技术大学附属第一医院精神科，生命科学与医学学院）
Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Multimedia content in social media platforms provides significant information during disaster events. The types of information shared include reports of injured or deceased people, infrastructure damage, and missing or found people, among others. Although many studies have shown the usefulness of both text and image content for disaster response purposes, the research has been mostly focused on analyzing only the text modality in the past. In this paper, we propose to use both text and image modalities of social media data to learn a joint representation using state-of-the-art deep learning te…
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School（图林根工业大学和汉诺威医学院医学信息学研究所）；Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)（下萨克森人工智能与因果医学中心（CAIMed））
Heterogeneous data fusion can enhance the robustness and accuracy of an algorithm on a given task. However, due to the difference in various modalities, aligning the sensors and embedding their information into discriminative and compact representations is challenging. In this paper, we propose a Contrastive learning based MultiModal Alignment Network (CoMMANet) to align data from different sensors into a shared and discriminative manifold where class information is preserved. The proposed architecture uses a multimodal triplet autoencoder to cluster the latent space in such a way that samples…
Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs
School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University（香港理工大学数据科学与人工智能系）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）
Agents that can follow language instructions are expected to be useful in a variety of situations such as navigation. However, training neural network-based agents requires numerous paired trajectories and languages. This paper proposes using multimodal generative models for semi-supervised learning in the instruction following tasks. The models learn a shared representation of the paired data, and enable semi-supervised learning by reconstructing unpaired data through the representation. Key challenges in applying the models to sequence-to-sequence tasks including instruction following are le…
Georgia Institute of Technology（佐治亚理工学院）；Arizona State University（亚利桑那州立大学）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）；MS Technologies Corporation（MS科技公司）
Inria（法国国家信息与自动化研究所）；Université de Bordeaux（波尔多大学）；Microsoft Research（微软研究院）
In this paper, we investigate whether artificial agents can develop a shared language in an ecological setting where communication relies on a sensory-motor channel. To this end, we introduce the Graphical Referential Game (GREG) where a speaker must produce a graphical utterance to name a visual referent object while a listener has to select the corresponding object among distractor referents, given the delivered message. The utterances are drawing images produced using dynamical motor primitives combined with a sketching library. To tackle GREG we present CURVES: a multimodal contrastive dee…
We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations). CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures. CoMIRs reduce the multimodal registration problem to a monomodal one, in which general intensity-based, as well as feature-based, registration algorithms can be applied. The method involves training one neural network per modality on aligned images, using a contrastive loss based on noise-contrasti…
Netease Fuxi AI Lab（网易伏羲AI实验室）；Hangzhou Dianzi University（杭州电子科技大学）
Humans are skilled in reading the interlocutor's emotion from multimodal signals, including spoken words, simultaneous speech, and facial expressions. It is still a challenge to effectively decode emotions from the complex interactions of multimodal signals. In this paper, we design three kinds of multimodal latent representations to refine the emotion analysis process and capture complex multimodal interactions from different views, including a intact three-modal integrating representation, a modality-shared representation, and three modality-individual representations. Then, a modality-seman…
MM-ACT: Learn from Multimodal Parallel Generation to Act
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）
