We present a conditional generative model that maps low-dimensional embeddings of multiple modalities of data to a common latent space hence extracting semantic relationships between them. The embedding specific to a modality is first extracted and subsequently a constrained optimization procedure is performed to project the two embedding spaces to a common manifold. The individual embeddings are generated back from this common latent space. However, in order to enable independent conditional inference for separately extracting the corresponding embeddings from the common latent space representation, we deploy a proxy variable trick - wherein, the single shared latent space is replaced by the respective separate latent spaces of each modality. We design an objective function, such that, during training we can force these separate spaces to lie close to each other, by minimizing the distance between their probability distribution functions. Experimental results demonstrate that the learned joint model can generalize to learning concepts of double MNIST digits with additional attributes of colors,from both textual and speech input.
Continuous dimensional emotion prediction is a challenging task where the fusion of various modalities usually achieves state-of-the-art performance such as early fusion or late fusion. In this paper, we propose a novel multi-modal fusion strategy named conditional attention fusion, which can dynamically pay attention to different modalities at each time step. Long-short term memory recurrent neural networks (LSTM-RNN) is applied as the basic uni-modality model to capture long time dependencies. The weights assigned to different modalities are automatically decided by the current input feature…
New York University（纽约大学）；NYU Grossman School of Medicine（纽约大学格罗斯曼医学院）；Courant Institute of Mathematical Sciences（库朗数学科学研究所）；Genentech（基因泰克）；CIFAR LMB（加拿大高等研究院分子生物学实验室）
We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks, these models tend to rely on just one modality while under-fitting the other modalities. Such behavior is counter-intuitive and hurts the models' generalization, as we observe empirically. To estimate the model's dependence on each modality, we compute the gain on the accuracy when the model has access to it in addition to another modality. We refer to this gain as the conditional utilization rate. In the experiments, we consistently observe an imbalance in conditional utilization rates between modali…
In this paper, we address the problem of conditional modality learning, whereby one is interested in generating one modality given the other. While it is straightforward to learn a joint distribution over multiple modalities using a deep multimodal architecture, we observe that such models aren't very effective at conditional generation. Hence, we address the problem by learning conditional distributions between the modalities. We use variational methods for maximizing the corresponding conditional log-likelihood. The resultant deep model, which we refer to as conditional multimodal autoencode…
Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation
Nanyang Technological University（南洋理工大学）；South China University of Technology（华南理工大学）；University of Oxford（牛津大学）；The University of Sydney（悉尼大学）；JD Explore Academy（京东探索研究院）
CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；Continental-NTU Corporate Lab, Nanyang Technological University（大陆集团-南洋理工大学联合企业实验室）；College of Computer Science and Technology, Harbin Engineering University（哈尔滨工程大学计算机科学与技术学院）
FeatInv: Spatially resolved mapping from feature space to input space using conditional diffusion models
Division AI4Health（AI4Health部门）；Carl von Ossietzky Universität Oldenburg（奥尔登堡卡尔·冯·奥西特齐克大学）；Explainable Artificial Intelligence Group（可解释人工智能小组）；Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因里希-赫兹研究所）
Feed-forward networks are widely used in cross-modal applications to bridge modalities by mapping distributed vectors of one modality to the other, or to a shared space. The predicted vectors are then used to perform e.g., retrieval or labeling. Thus, the success of the whole system relies on the ability of the mapping to make the neighborhood structure (i.e., the pairwise similarities) of the predicted vectors akin to that of the target vectors. However, whether this is achieved has not been investigated yet. Here, we propose a new similarity measure and two ad hoc experiments to shed light o…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
Prediction of future states of the environment and interacting agents is a key competence required for autonomous agents to operate successfully in the real world. Prior work for structured sequence prediction based on latent variable models imposes a uni-modal standard Gaussian prior on the latent variables. This induces a strong model bias which makes it challenging to fully capture the multi-modality of the distribution of the future states. In this work, we introduce Conditional Flow Variational Autoencoders (CF-VAE) using our novel conditional normalizing flow based prior to capture compl…
CNRS-ISIR, Sorbonne University（索邦大学法国国家科学研究中心智能系统与机器人研究所）；Criteo AI Lab（科韬人工智能实验室）；Université de Rouen, LITIS（鲁昂大学LITIS实验室）
We consider the problem of unsupervised domain adaptation (UDA) between a source and a target domain under conditional and label shift a.k.a Generalized Target Shift (GeTarS). Unlike simpler UDA settings, few works have addressed this challenging problem. Recent approaches learn domain-invariant representations, yet they have practical limitations and rely on strong assumptions that may not hold in practice. In this paper, we explore a novel and general approach to align pretrained representations, which circumvents existing drawbacks. Instead of constraining representation invariance, it lear…
BI Norwegian Business School（BI挪威商学院）；UiT The Arctic University of Norway（挪威北极大学）；Norwegian Computing Center（挪威计算中心）
Deep generative models with latent variables have been used lately to learn joint representations and generative processes from multi-modal data. These two learning mechanisms can, however, conflict with each other and representations can fail to embed information on the data modalities. This research studies the realistic scenario in which all modalities and class labels are available for model training, but where some modalities and labels required for downstream tasks are missing. We show, in this scenario, that the variational lower bound limits mutual information between joint representat…
The University of Hull（赫尔大学）；University of Sunderland（桑德兰大学）
Forecasting short-term motion of nearby vehicles presents an inherently challenging issue as the space of their possible future movements is not strictly limited to a set of single trajectories. Recently proposed techniques that demonstrate plausible results concentrate primarily on forecasting a fixed number of deterministic predictions, or on classifying over a wide variety of trajectories that were previously generated using e.g. dynamic model. This paper focuses on addressing the uncertainty associated with the discussed task by utilising the stochastic nature of generative models in order…
Univ. of Alberta（阿尔伯塔大学）；Amii（阿尔伯塔机器智能研究院）；Vector Institute（向量研究所）；Univ. of Toronto（多伦多大学）
For multi-valued functions---such as when the conditional distribution on targets given the inputs is multi-modal---standard regression approaches are not always desirable because they provide the conditional mean. Modal regression algorithms address this issue by instead finding the conditional mode(s). Most, however, are nonparametric approaches and so can be difficult to scale. Further, parametric approximators, like neural networks, facilitate learning complex relationships between inputs and targets. In this work, we propose a parametric modal regression algorithm. We use the implicit fun…
ZeroFlood: Flood Hazard Mapping from Single-Modality SAR Using Geo-Foundation Models
German Aerospace Center (DLR), Remote Sensing Technology Institute（德国航空航天中心（DLR）遥感技术研究所）；ETH Zurich, Seminar for Applied Mathematics（苏黎世联邦理工学院应用数学研讨会）；ETH AI Center（苏黎世联邦理工学院人工智能中心）
Bipartite Patient-Modality Graph Learning with Event-Conditional Modelling of Censoring for Cancer Survival Prediction
Hunan Provincial Key Lab on Bioinformatics, School of Computer Science and Engineering, Central South University（湖南省级生物信息学重点实验室，计算机科学与工程学院，中南大学）；Xinjiang Engineering Research Center of Big Data and Intelligent Software, School of Software, Xinjiang University（新疆大数据与智能软件工程研究中心，软件学院，新疆大学）
Large scale pretrained language models have demonstrated state-of-the-art performance in language understanding tasks. Their application has recently expanded into multimodality learning, leading to improved representations combining vision and language. However, progress in adapting language models towards conditional Natural Language Generation (NLG) has been limited to a single modality, generally text. We propose MAnTiS, Multimodal Adaptation for Text Synthesis, a general approach for multimodal conditionality in transformer-based NLG models. In this method, we pass inputs from each modali…
Artificial Intelligence Department, Faculty of Computer Engineering, University of Isfahan（伊斯法罕大学计算机工程学院人工智能系）；Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab（格勒诺布尔阿尔卑斯大学（Univ. Grenoble Alpes）、法国国家科学研究中心（CNRS）、格勒诺布尔综合理工学院（Grenoble INP）、GIPSA实验室（GIPSA-lab））
Multimodal manifold modeling methods extend the spectral geometry-aware data analysis to learning from several related and complementary modalities. Most of these methods work based on two major assumptions: 1) there are the same number of homogeneous data samples in each modality, and 2) at least partial correspondences between modalities are given in advance as prior knowledge. This work proposes two new multimodal modeling methods. The first method establishes a general analyzing framework to deal with the multimodal information problem for heterogeneous data without any specific prior know…
Johannes Kepler University Linz（约翰内斯开普勒大学林茨分校）；Institute of Computational Perception（计算感知研究所）；LIT Artificial Intelligence Lab（LIT人工智能实验室）
This paper addresses the problem of sheet-image-based on-line audio-to-score alignment also known as score following. Drawing inspiration from object detection, a conditional neural network architecture is proposed that directly predicts x,y coordinates of the matching positions in a complete score sheet image at each point in time for a given musical performance. Experiments are conducted on a synthetic polyphonic piano benchmark dataset and the new method is compared to several existing approaches from the literature for sheet-image-based score following as well as an Optical Music Recogniti…
The increasing intensity and frequency of floods is one of the many consequences of our changing climate. In this work, we explore ML techniques that improve the flood detection module of an operational early flood warning system. Our method exploits an unlabelled dataset of paired multi-spectral and Synthetic Aperture Radar (SAR) imagery to reduce the labeling requirements of a purely supervised learning method. Prior works have used unlabelled data by creating weak labels out of them. However, from our experiments we noticed that such a model still ends up learning the label mistakes in thos…
RIKEN Center for Advance Intelligence Project（理化学研究所革新智能研究中心）；the University of Tokyo（东京大学）；Hiroshima University（广岛大学）；Chiba University（千叶大学）；Tokyo Institute of Technology（东京工业大学）；Tohoku University（东北大学）
Earth observation technologies, such as optical imaging and synthetic aperture radar (SAR), provide excellent means to monitor ever-growing urban environments continuously. Notably, in the case of large-scale disasters (e.g., tsunamis and earthquakes), in which a response is highly time-critical, images from both data modalities can complement each other to accurately convey the full damage condition in the disaster's aftermath. However, due to several factors, such as weather and satellite coverage, it is often uncertain which data modality will be the first available for rapid disaster respo…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）
Data from multi-modality provide complementary information in clinical prediction, but missing data in clinical cohorts limits the number of subjects in multi-modal learning context. Multi-modal missing imputation is challenging with existing methods when 1) the missing data span across heterogeneous modalities (e.g., image vs. non-image); or 2) one modality is largely missing. In this paper, we address imputation of missing data by modeling the joint distribution of multi-modal data. Motivated by partial bidirectional generative adversarial net (PBiGAN), we propose a new Conditional PBiGAN (C…
Université de Sousse（苏塞大学）；Ecole Nationale d’Ingénieurs de Sousse（苏塞国立工程师学院）；Istanbul Technical University（伊斯坦布尔理工大学）
Brain graph synthesis marked a new era for predicting a target brain graph from a source one without incurring the high acquisition cost and processing time of neuroimaging data. However, existing multi-modal graph synthesis frameworks have several limitations. First, they mainly focus on generating graphs from the same domain (intra-modality), overlooking the rich multimodal representations of brain connectivity (inter-modality). Second, they can only handle isomorphic graph generation tasks, limiting their generalizability to synthesizing target graphs with a different node size and topologi…
We introduce a model for bidirectional retrieval of images and sentences through a multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. In addition to a ranking objective seen in previous work, this allows us to add a new fragment alignment objective that learns to directly associate these fragments across modalities. Extensive experimental eva…
Future University Hakodate（公立函馆未来大学）；NEC Corporation（日本电气股份有限公司）；Tokyo Institute of Technology（东京工业大学）；RIKEN AIP（理化学研究所AIP）
Modal regression is aimed at estimating the global mode (i.e., global maximum) of the conditional density function of the output variable given input variables, and has led to regression methods robust against heavy-tailed or skewed noises. The conditional mode is often estimated through maximization of the modal regression risk (MRR). In order to apply a gradient method for the maximization, the fundamental challenge is accurate approximation of the gradient of MRR, not MRR itself. To overcome this challenge, in this paper, we take a novel approach of directly approximating the gradient of MR…
We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative Adversarial Network (cGAN). Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code. To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes. The proposed regularization is simple, general, and can be easily integrate…
