Spoken question answering (SQA) requires fine-grained understanding of both spoken documents and questions for the optimal answer prediction. In this paper, we propose novel training schemes for spoken question answering with a self-supervised training stage and a contrastive representation learning stage. In the self-supervised stage, we propose three auxiliary self-supervised tasks, including utterance restoration, utterance insertion, and question discrimination, and jointly train the model to capture consistency and coherence among speech documents without any additional data or annotations. We then propose to learn noise-invariant utterance representations in a contrastive objective by adopting multiple augmentation strategies, including span deletion and span substitution. Besides, we design a Temporal-Alignment attention to semantically align the speech-text clues in the learned common space and benefit the SQA tasks. By this means, the training schemes can more effectively guide the generation model to predict more proper answers. Experimental results show that our model achieves state-of-the-art results on three SQA benchmarks.
Multimodal sentiment analysis has become an increasingly popular research area as the demand for multimodal online content is growing. For multimodal sentiment analysis, words can have different meanings depending on the linguistic context and non-verbal information, so it is crucial to understand the meaning of the words accordingly. In addition, the word meanings should be interpreted within the whole utterance context that includes nonverbal information. In this paper, we present a Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representatio…
We present modality gap, an intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) are embedded at arm's length in their shared representation in multi-modal models such as CLIP. Our systematic analysis demonstrates that this gap is caused by a combination of model initialization and contrastive learning optimization. In model initialization, we show empirically and theoretically that the representation of a common deep neural network is restricted to a narrow cone. As a consequence, in a mu…
PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset
Department of Computer Science（计算机科学系）；Brown University（布朗大学）；Institute for Bioinformatics and Medical Informatics（生物信息学与医学信息学研究所）；University of Tübingen（图宾根大学）；Center for Computational Molecular Biology（计算分子生物学中心）
New York University（纽约大学）；Hyperfine Research（Hyperfine研究公司）；Yale University（耶鲁大学）
Establishing voxelwise semantic correspondence across distinct imaging modalities is a foundational yet formidable computer vision task. Current multi-modality registration techniques maximize hand-crafted inter-domain similarity functions, are limited in modeling nonlinear intensity-relationships and deformations, and may require significant re-engineering or underperform on new tasks, datasets, and domain pairs. This work presents ContraReg, an unsupervised contrastive representation learning approach to multi-modality deformable registration. By projecting learned multi-scale local patch fe…
Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Hebei University of Science and Technology（河北科技大学）
RMIT University（皇家墨尔本理工大学）；University of New South Wales（新南威尔士大学）；Philips Research（飞利浦研究院）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）
Self-Supervised Learning (SSL) is a new paradigm for learning discriminative representations without labelled data and has reached comparable or even state-of-the-art results in comparison to supervised counterparts. Contrastive Learning (CL) is one of the most well-known approaches in SSL that attempts to learn general, informative representations of data. CL methods have been mostly developed for applications in computer vision and natural language processing where only a single sensor modality is used. A majority of pervasive computing applications, however, exploit data from a range of dif…
Multi-modal entity alignment aims to identify equivalent entities between two different multi-modal knowledge graphs, which consist of structural triples and images associated with entities. Most previous works focus on how to utilize and encode information from different modalities, while it is not trivial to leverage multi-modal knowledge in entity alignment because of the modality heterogeneity. In this paper, we propose MCLEA, a Multi-modal Contrastive Learning based Entity Alignment model, to obtain effective joint representations for multi-modal entity alignment. Different from previous …
SaliencyI2PLoc: saliency-guided image-point cloud localization using contrastive learning
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing（测绘遥感信息工程国家重点实验室）；Wuhan University（武汉大学）；School of Electrical and Electronic Engineering（电气与电子工程学院）；Nanyang Technological University（南洋理工大学）；School of Geography and Environment（地理与环境学院）；Jiangxi Normal University（江西师范大学）
Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning
School of Computer Science and Engineering, Central South University（计算机科学与工程学院，中南大学）；School of Automation, Central South University（自动化学院，中南大学）；School of Computer Science and Technology, Beijing Jiaotong University（计算机科学与技术学院，北京交通大学）
Cadence Design Systems (India) Private Limited（楷登电子科技（印度）私人有限公司）；Indian Institute of Technology, Jodhpur（焦特布尔印度理工学院）
Contrastive learning is a powerful technique to learn representations that are semantically distinctive and geometrically invariant. While most of the earlier approaches have demonstrated its effectiveness on single-modality learning tasks such as image classification, recently there have been a few attempts towards extending this idea to multi-modal data. In this paper, we propose two loss functions based on normalized cross-entropy to perform the task of learning joint visual-semantic embedding using batch contrastive training. In a batch, for a given anchor point from one modality, we consi…
UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing Jiaotong University（北京交通大学）；Northwestern Polytechnical University（西北工业大学）；Alibaba Group, US（美国阿里巴巴集团）；National University of Singapore（新加坡国立大学）
KTH Royal Institute of Technology（KTH皇家理工学院）；INESC-ID（葡萄牙系统与计算机工程研究所）；Instituto Superior Técnico, University of Lisbon（里斯本大学高等理工学院）
Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method consisting of two main components: i) a two-level architecture consisting of modality-specific base encoders, allowing to process an arbitrary number of modalities to an intermediate representation of fixed dimensionality, and a shared projection head, mapping the intermedi…
When people observe events, they are able to abstract key information and build concise summaries of what is happening. These summaries include contextual and semantic information describing the important high-level details (what, where, who and how) of the observed event and exclude background information that is deemed unimportant to the observer. With this in mind, the descriptions people generate for videos of different dynamic events can greatly improve our understanding of the key information of interest in each video. These descriptions can be captured in captions that provide expanded …
Multi-Modal Contrastive Learning for Implicit Earth Embeddings via Location Tying
Computational Methods Lab, HafenCity University Hamburg（汉堡港城大学计算方法实验室）；Dept. of Operations & Technology, Technical University of Munich（慕尼黑工业大学运营与技术系；海尔布隆数据科学中心；慕尼黑数据科学研究所）；Heilbronn Data Science Center（波恩大学大地测量与地理信息研究所）；Munich Data Science Institute；Institute of Geodesy and Geoinformation, University of Bonn
Spoken language understanding (SLU) is an essential task for machines to understand human speech for better interactions. However, errors from the automatic speech recognizer (ASR) usually hurt the understanding performance. In reality, ASR systems may not be easy to adjust for the target scenarios. Therefore, this paper focuses on learning utterance representations that are robust to ASR errors using a contrastive objective, and further strengthens the generalization ability by combining supervised contrastive learning and self-distillation in model fine-tuning. Experiments on three benchmark…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University at Buffalo（布法罗大学）；Amazon（亚马逊）
Contrastive loss has been increasingly used in learning representations from multiple modalities. In the limit, the nature of the contrastive loss encourages modalities to exactly match each other in the latent space. Yet it remains an open question how the modality alignment affects the downstream task performance. In this paper, based on an information-theoretic argument, we first prove that exact modality alignment is sub-optimal in general for downstream prediction tasks. Hence we advocate that the key of better performance lies in meaningful latent modality structures instead of perfect m…
In this paper, we explore the unsupervised learning of a semantic embedding space for co-occurring sensory inputs. Specifically, we focus on the task of learning a semantic vector space for both spoken and handwritten digits using the TIDIGITs and MNIST datasets. Current techniques encode image and audio/textual inputs directly to semantic embeddings. In contrast, our technique maps an input to the mean and log variance vectors of a diagonal Gaussian from which sample semantic embeddings are drawn. In addition to encouraging semantic similarity between co-occurring inputs,our loss function inc…
Multimodal Federated Learning via Contrastive Representation Ensemble
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Waterloo（滑铁卢大学）；Carnegie Mellon University（卡内基梅隆大学）
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）
Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin（柏林健康研究所，柏林查理医院）；Intelligent Medicine Institute, Fudan University（智能医学研究院，复旦大学）；Department of Mathematics and Computer Science, Freie Universität Berlin（数学与计算机科学系，柏林自由大学）
HC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding
University of Science and Technology Beijing（北京科技大学）；Agency for Science, Technology and Research（新加坡科技研究局）；Nanyang Technological University（南洋理工大学）；University of Technology Sydney（悉尼科技大学）
Despite the great success of spoken language understanding (SLU) in high-resource languages, it remains challenging in low-resource languages mainly due to the lack of labeled training data. The recent multilingual code-switching approach achieves better alignments of model representations across languages by constructing a mixed-language context in zero-shot cross-lingual SLU. However, current code-switching methods are limited to implicit alignment and disregard the inherent semantic structure in SLU, i.e., the hierarchical inclusion of utterances, slots, and words. In this paper, we propose…
COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Centre for Vision, Speech, and Signal Processing (CVSSP), University of Surrey（Surrey 大学视觉、语音和信号处理中心）
Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Research Asia（微软亚洲研究院）；National University of Singapore（新加坡国立大学）
Due to high data demands of current methods, attention to zero-shot cross-lingual spoken language understanding (SLU) has grown, as such approaches greatly reduce human annotation effort. However, existing models solely rely on shared parameters, which can only perform implicit alignment across languages. We present Global--Local Contrastive Learning Framework (GL-CLeF) to address this shortcoming. Specifically, we employ contrastive learning, leveraging bilingual dictionaries to construct multilingual views of the same utterance, then encourage their representations to be more similar than ne…
