Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, a line of recent work reveals that contrastive learning can invert the data generating process and recover ground truth latent factors shared between views. In this work, we present new identifiability results for multimodal contrastive learning, showing that it is possible to recover shared factors in a more general setup than the multi-view setting studied previously. Specifically, we distinguish between the multi-view setting with one generative mechanism (e.g., multiple cameras of the same type) and the multimodal setting that is characterized by distinct mechanisms (e.g., cameras and microphones). Our work generalizes previous identifiability results by redefining the generative process in terms of distinct mechanisms with modality-specific latent variables. We prove that contrastive learning can block-identify latent factors shared between modalities, even when there are nontrivial dependencies between factors. We empirically verify our identifiability results with numerical simulations and corroborate our findings on a complex multimodal dataset of image/text pairs. Zooming out, our work provides a theoretical basis for multimodal representation learning and explains in which settings multimodal contrastive learning can be effective in practice.
Nonlinear independent component analysis (nICA) aims at recovering statistically independent latent components that are mixed by unknown nonlinear functions. Central to nICA is the identifiability of the latent components, which had been elusive until very recently. Specifically, Hyvärinen et al. have shown that the nonlinearly mixed latent components are identifiable (up to often inconsequential ambiguities) under a generalized contrastive learning (GCL) formulation, given that the latent components are independent conditioned on a certain auxiliary variable. The GCL-based identifiability of …
The Gatsby Unit UCL（伦敦大学学院盖茨比研究组）；Dept. of CS and HIIT Univ. Helsinki（赫尔辛基大学计算机科学系及赫尔辛基信息技术研究所）；Div. of Info. Sci. NAIST（奈良先端科学技术大学院大学信息科学部）；Univ. Cambridge & Microsoft Research（剑桥大学及微软研究院）
Nonlinear ICA is a fundamental problem for unsupervised representation learning, emphasizing the capacity to recover the underlying latent variables generating the data (i.e., identifiability). Recently, the very first identifiability proofs for nonlinear ICA have been proposed, leveraging the temporal structure of the independent components. Here, we propose a general framework for nonlinear ICA, which, as a special case, can make use of temporal structure. It is based on augmenting the data by an auxiliary variable, such as the time index, the history of the time series, or any other availab…
Contrastive Learning on Multimodal Analysis of Electronic Health Records
Harvard T.H. Chan School of Public Health（哈佛T.H. 柏林公共卫生学院）；Harvard Medical School（哈佛医学院）；Rutgers University（罗格斯大学）；National University of Singapore（新加坡国立大学）
Language-supervised vision models have recently attracted great attention in computer vision. A common approach to build such models is to use contrastive learning on paired data across the two modalities, as exemplified by Contrastive Language-Image Pre-Training (CLIP). In this paper, under linear representation settings, (i) we initiate the investigation of a general class of nonlinear loss functions for multimodal contrastive learning (MMCL) including CLIP loss and show its connection to singular value decomposition (SVD). Namely, we show that each step of loss minimization by gradient desc…
Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our learning principle, time-contrastive learning (TCL), finds a representation which allows optimal discrimination of time segments (windows). Surprisingly, we show how TCL can be related to a nonlinear ICA model, when ICA is redefined to include temporal nonstationarities. In partic…
KTH Royal Institute of Technology（KTH皇家理工学院）；INESC-ID（葡萄牙系统与计算机工程研究所）；Instituto Superior Técnico, University of Lisbon（里斯本大学高等理工学院）
Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method consisting of two main components: i) a two-level architecture consisting of modality-specific base encoders, allowing to process an arbitrary number of modalities to an intermediate representation of fixed dimensionality, and a shared projection head, mapping the intermedi…
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin（柏林健康研究院，柏林夏里特大学医学中心）；Department of Mathematics and Computer Science, Freie Universität Berlin（柏林自由大学数学与计算机科学系）；Intelligent Medicine Institute, Fudan University（复旦大学智能医学研究院）
Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning
Berlin Institute of Health（柏林卫生研究所）；Charité - Universitätsmedizin Berlin（柏林夏里特医学院）；Freie Universität Berlin（柏林自由大学）；Intelligent Medicine Institute（智能医学研究所）；Fudan University（复旦大学）
Local-Global Multimodal Contrastive Learning for Molecular Property Prediction
School of Mathematical Sciences（数学科学学院）；University of Electronic Science and Technology of China（电子科技大学）；Department of Computer Science and Engineer（计算机科学与工程系）；Oakland University（奥克兰大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；College of Management Science（管理科学学院）；Chengdu University of Technology（成都理工大学）
Multimodal imaging and correlative analysis typically require image alignment. Contrastive learning can generate representations of multimodal images, reducing the challenging task of multimodal image registration to a monomodal one. Previously, additional supervision on intermediate layers in contrastive learning has improved biomedical image classification. We evaluate if a similar approach improves representations learned for registration to boost registration performance. We explore three approaches to add contrastive supervision to the latent features of the bottleneck layer in the U-Nets…
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School（图林根工业大学和汉诺威医学院医学信息学研究所）；Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)（下萨克森人工智能与因果医学中心（CAIMed））
Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin（柏林健康研究所，柏林查理医院）；Intelligent Medicine Institute, Fudan University（智能医学研究院，复旦大学）；Department of Mathematics and Computer Science, Freie Universität Berlin（数学与计算机科学系，柏林自由大学）
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos -- while preserving the transitivity across these modalities. We explore several new IMU-based applications that IMU2CLIP enables, such as motion-based media retrieval and natural language reasoning tasks …
Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Hebei University of Science and Technology（河北科技大学）
Advertising click fraud detection plays one of the vital roles in current E-commerce websites as advertising is an essential component of its business model. It aims at, given a set of corresponding features, e.g., demographic information of users and statistical features of clicks, predicting whether a click is fraudulent or not in the community. Recent efforts attempted to incorporate attributed behavior sequence and heterogeneous network for extracting complex features of users and achieved significant effects on click fraud detection. In this paper, we propose a Multimodal and Contrastive …
Self-supervised methods have shown tremendous success in the field of computer vision, including applications in remote sensing and medical imaging. Most popular contrastive-loss based methods like SimCLR, MoCo, MoCo-v2 use multiple views of the same image by applying contrived augmentations on the image to create positive pairs and contrast them with negative examples. Although these techniques work well, most of these techniques have been tuned on ImageNet (and similar computer vision datasets). While there have been some attempts to capture a richer set of deformations in the positive sampl…
An Enhanced Dual Transformer Contrastive Network for Multimodal Sentiment Analysis
Ho Chi Minh City Open University, Ho Chi Minh, Vietnam Insight Centre for Data Analytics \& School of Computing, Dublin City University, Dublin, Ireland
Creating artificial social intelligence - algorithms that can understand the nuances of multi-person interactions - is an exciting and emerging challenge in processing facial expressions and gestures from multimodal videos. Recent multimodal methods have set the state of the art on many tasks, but have difficulty modeling the complex face-to-face conversational dynamics across speaking turns in social interaction, particularly in a self-supervised setup. In this paper, we propose Face-to-Face Contrastive Learning (F2F-CL), a graph neural network designed to model social interactions using fact…
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Department of Computer Science and Engineering（计算机科学与工程系）；Incheon National University（庆尚国立大学）；Department of Artificial Intelligence（人工智能系）；Inha University（Inha大学）；Center for Brain-Machine Interface（脑机接口中心）
Hasso Plattner Institute for Digital Engineering, University of Potsdam（波茨坦大学哈索·普拉特纳数字工程研究所）；TU Kaiserslautern（凯泽斯劳滕工业大学）；Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院哈索·普拉特纳数字健康研究所）
High annotation costs are a substantial bottleneck in applying modern deep learning architectures to clinically relevant medical use cases, substantiating the need for novel algorithms to learn from unlabeled data. In this work, we propose ContIG, a self-supervised method that can learn from large datasets of unlabeled medical images and genetic data. Our approach aligns images and several genetic modalities in the feature space using a contrastive loss. We design our method to integrate multiple modalities of each individual person in the same model end-to-end, even when the available modalit…
Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU
Aalborg University（奥尔堡大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Milestone Systems（里程碑系统公司）；Pioneer Center for Artificial Intelligence（先锋人工智能中心）
