Contrastive learning has become one of the most impressive approaches for multi-modal representation learning. However, previous multi-modal works mainly focused on cross-modal understanding, ignoring in-modal contrastive learning, which limits the representation of each modality. In this paper, we propose a novel contrastive learning strategy, called $Turbo$, to promote multi-modal understanding by joint in-modal and cross-modal contrastive learning. Specifically, multi-modal data pairs are sent through the forward pass twice with different hidden dropout masks to get two different representations for each modality. With these representations, we obtain multiple in-modal and cross-modal contrastive objectives for training. Finally, we combine the self-supervised Turbo with the supervised multi-modal classification and demonstrate its effectiveness on two audio-text classification tasks, where the state-of-the-art performance is achieved on a speech emotion recognition benchmark dataset.
Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Hebei University of Science and Technology（河北科技大学）
On the Comparison between Multi-modal and Single-modal Contrastive Learning
RIKEN AIP（理化学研究所人工智能项目）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；MBZUAI（Mohamed bin Zayed 人工智能大学）；University of Tokyo（东京大学）
Multi-modal entity alignment aims to identify equivalent entities between two different multi-modal knowledge graphs, which consist of structural triples and images associated with entities. Most previous works focus on how to utilize and encode information from different modalities, while it is not trivial to leverage multi-modal knowledge in entity alignment because of the modality heterogeneity. In this paper, we propose MCLEA, a Multi-modal Contrastive Learning based Entity Alignment model, to obtain effective joint representations for multi-modal entity alignment. Different from previous …
PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset
Department of Computer Science（计算机科学系）；Brown University（布朗大学）；Institute for Bioinformatics and Medical Informatics（生物信息学与医学信息学研究所）；University of Tübingen（图宾根大学）；Center for Computational Molecular Biology（计算分子生物学中心）
We present modality gap, an intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) are embedded at arm's length in their shared representation in multi-modal models such as CLIP. Our systematic analysis demonstrates that this gap is caused by a combination of model initialization and contrastive learning optimization. In model initialization, we show empirically and theoretically that the representation of a common deep neural network is restricted to a narrow cone. As a consequence, in a mu…
Multi-Modal Contrastive Learning for Implicit Earth Embeddings via Location Tying
Computational Methods Lab, HafenCity University Hamburg（汉堡港城大学计算方法实验室）；Dept. of Operations & Technology, Technical University of Munich（慕尼黑工业大学运营与技术系；海尔布隆数据科学中心；慕尼黑数据科学研究所）；Heilbronn Data Science Center（波恩大学大地测量与地理信息研究所）；Munich Data Science Institute；Institute of Geodesy and Geoinformation, University of Bonn
Cadence Design Systems (India) Private Limited（楷登电子科技（印度）私人有限公司）；Indian Institute of Technology, Jodhpur（焦特布尔印度理工学院）
Contrastive learning is a powerful technique to learn representations that are semantically distinctive and geometrically invariant. While most of the earlier approaches have demonstrated its effectiveness on single-modality learning tasks such as image classification, recently there have been a few attempts towards extending this idea to multi-modal data. In this paper, we propose two loss functions based on normalized cross-entropy to perform the task of learning joint visual-semantic embedding using batch contrastive training. In a batch, for a given anchor point from one modality, we consi…
Various state-of-the-art self-supervised visual representation learning approaches take advantage of data from multiple sensors by aligning the feature representations across views and/or modalities. In this work, we investigate how aligning representations affects the visual features obtained from cross-view and cross-modal contrastive learning on images and point clouds. On five real-world datasets and on five tasks, we train and evaluate 108 models based on four pretraining variations. We find that cross-modal representation alignment discards complementary visual information, such as color…
Contrastive learning allows us to flexibly define powerful losses by contrasting positive pairs from sets of negative samples. Recently, the principle has also been used to learn cross-modal embeddings for video and text, yet without exploiting its full potential. In particular, previous losses do not take the intra-modality similarities into account, which leads to inefficient embeddings, as the same content is mapped to multiple points in the embedding space. With CrossCLR, we present a contrastive loss that fixes this issue. Moreover, we define sets of highly related samples in terms of the…
New York University（纽约大学）；Hyperfine Research（Hyperfine研究公司）；Yale University（耶鲁大学）
Establishing voxelwise semantic correspondence across distinct imaging modalities is a foundational yet formidable computer vision task. Current multi-modality registration techniques maximize hand-crafted inter-domain similarity functions, are limited in modeling nonlinear intensity-relationships and deformations, and may require significant re-engineering or underperform on new tasks, datasets, and domain pairs. This work presents ContraReg, an unsupervised contrastive representation learning approach to multi-modality deformable registration. By projecting learned multi-scale local patch fe…
Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin（柏林健康研究所，柏林查理医院）；Intelligent Medicine Institute, Fudan University（智能医学研究院，复旦大学）；Department of Mathematics and Computer Science, Freie Universität Berlin（数学与计算机科学系，柏林自由大学）
Improving Paratope and Epitope Prediction by Multi-Modal Contrastive Learning and Interaction Informativeness Estimation
College of Informatics, Huazhong Agricultural University（华中农业大学信息学院）；Hubei Key Laboratory of Agricultural Bioinformatics, Huazhong Agricultural University（华中农业大学湖北省农业生物信息重点实验室）；Engineering Research Center of Intelligent Technology for Agriculture, Ministry of Education（教育部农业智能技术工程研究中心）
University of Tübingen（蒂宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；Tencent PCG（腾讯平台与内容事业群）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Having access to multi-modal cues (e.g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality. In this work, we propose to transfer knowledge across heterogeneous modalities, even though these data modalities may not be semantically correlated. Rather than directly aligning the representations of different modalities, we compose audio, image, and video representations across modalities to uncover richer multi-modal knowledge. Our main idea is to learn a compositional embedding that closes the cross-modal semantic gap and captures the task…
Multimodal sentiment analysis has become an increasingly popular research area as the demand for multimodal online content is growing. For multimodal sentiment analysis, words can have different meanings depending on the linguistic context and non-verbal information, so it is crucial to understand the meaning of the words accordingly. In addition, the word meanings should be interpreted within the whole utterance context that includes nonverbal information. In this paper, we present a Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representatio…
RMIT University（皇家墨尔本理工大学）；University of New South Wales（新南威尔士大学）；Philips Research（飞利浦研究院）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）
Self-Supervised Learning (SSL) is a new paradigm for learning discriminative representations without labelled data and has reached comparable or even state-of-the-art results in comparison to supervised counterparts. Contrastive Learning (CL) is one of the most well-known approaches in SSL that attempts to learn general, informative representations of data. CL methods have been mostly developed for applications in computer vision and natural language processing where only a single sensor modality is used. A majority of pervasive computing applications, however, exploit data from a range of dif…
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
Facial micro-expressions recognition has attracted much attention recently. Micro-expressions have the characteristics of short duration and low intensity, and it is difficult to train a high-performance classifier with the limited number of existing micro-expressions. Therefore, recognizing micro-expressions is a challenge task. In this paper, we propose a micro-expression recognition method based on attribute information embedding and cross-modal contrastive learning. We use 3D CNN to extract RGB features and FLOW features of micro-expression sequences and fuse them, and use BERT network to …
KTH Royal Institute of Technology（KTH皇家理工学院）；INESC-ID（葡萄牙系统与计算机工程研究所）；Instituto Superior Técnico, University of Lisbon（里斯本大学高等理工学院）
Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method consisting of two main components: i) a two-level architecture consisting of modality-specific base encoders, allowing to process an arbitrary number of modalities to an intermediate representation of fixed dimensionality, and a shared projection head, mapping the intermedi…
Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series
Shanghai Jiao Tong University（上海交通大学）；Key Laboratory for Thin Film and Microfabrication of Ministry of Education（教育部薄膜与微细加工重点实验室）；Guangdong JiuZhi Technology Co., Ltd（广东九智科技有限公司）；Shenzhen University（深圳大学）
Group Contrastive Learning for Weakly Paired Multimodal Data
Research and Early Development (gRED), Genentech（Genentech 研究与早期发展部门）；UCLA（加州大学洛杉矶分校）；Biology Research | AI Development (BRAID), Genentech（Genentech 生物研究 | 人工智能开发部门）；Genentech Computational Sciences NYU（Genentech 计算科学与纽约大学）
ETSCL: An Evidence Theory-Based Supervised Contrastive Learning Framework for Multi-modal Glaucoma Grading
Beihang University（北京航空航天大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李光前医学院）；Sun Yat-sen University（中山大学）；National University of Singapore（新加坡国立大学）；South China University of Technology（华南理工大学）
Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task,…
Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. Specifically, we show that many test instances are either over- or under-represented during retrieval, significantly hurting the retrieval performance. To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise bi…
Hasso Plattner Institute for Digital Engineering, University of Potsdam（波茨坦大学哈索·普拉特纳数字工程研究所）；TU Kaiserslautern（凯泽斯劳滕工业大学）；Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院哈索·普拉特纳数字健康研究所）
High annotation costs are a substantial bottleneck in applying modern deep learning architectures to clinically relevant medical use cases, substantiating the need for novel algorithms to learn from unlabeled data. In this work, we propose ContIG, a self-supervised method that can learn from large datasets of unlabeled medical images and genetic data. Our approach aligns images and several genetic modalities in the feature space using a contrastive loss. We design our method to integrate multiple modalities of each individual person in the same model end-to-end, even when the available modalit…
Spoken question answering (SQA) requires fine-grained understanding of both spoken documents and questions for the optimal answer prediction. In this paper, we propose novel training schemes for spoken question answering with a self-supervised training stage and a contrastive representation learning stage. In the self-supervised stage, we propose three auxiliary self-supervised tasks, including utterance restoration, utterance insertion, and question discrimination, and jointly train the model to capture consistency and coherence among speech documents without any additional data or annotation…
FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization
Hanoi University of Science and Technology（河内科学技术大学）；Nagoya University（名古屋大学）；Washington State University（华盛顿州立大学）；Swiss Federal Institute of Technology Lausanne（洛桑联邦理工学院）
There are a wide range of applications that involve multi-modal data, such as cross-modal retrieval, visual question-answering, and image captioning. Such applications are primarily dependent on aligned distributions of the different constituent modalities. Existing approaches generate latent embeddings for each modality in a joint fashion by representing them in a common manifold. However these joint embedding spaces fail to sufficiently reduce the modality gap, which affects the performance in downstream tasks. We hypothesize that these embeddings retain the intra-class relationships but are…
