We present Contrastive Neighborhood Alignment (CNA), a manifold learning approach to maintain the topology of learned features whereby data points that are mapped to nearby representations by the source (teacher) model are also mapped to neighbors by the target (student) model. The target model aims to mimic the local structure of the source representation space using a contrastive loss. CNA is an unsupervised learning algorithm that does not require ground-truth labels for the individual samples. CNA is illustrated in three scenarios: manifold learning, where the model maintains the local topology of the original data in a dimension-reduced space; model distillation, where a small student model is trained to mimic a larger teacher; and legacy model update, where an older model is replaced by a more powerful one. Experiments show that CNA is able to capture the manifold in a high-dimensional space and improves performance compared to the competing methods in their domains.
Entity alignment aims at integrating heterogeneous knowledge from different knowledge graphs. Recent studies employ embedding-based methods by first learning the representation of Knowledge Graphs and then performing entity alignment via measuring the similarity between entity embeddings. However, they failed to make good use of the relation semantic information due to the trade-off problem caused by the different objectives of learning knowledge embedding and neighborhood consensus. To address this problem, we propose Relational Knowledge Distillation for Entity Alignment (RKDEA), a Graph Con…
MIT（麻省理工学院）；MSU（密歇根州立大学）；IBM Research AI（IBM研究院人工智能中心）；UCSD（加州大学圣地亚哥分校）
As a seminal tool in self-supervised representation learning, contrastive learning has gained unprecedented attention in recent years. In essence, contrastive learning aims to leverage pairs of positive and negative samples for representation learning, which relates to exploiting neighborhood information in a feature space. By investigating the connection between contrastive learning and neighborhood component analysis (NCA), we provide a novel stochastic nearest neighbor viewpoint of contrastive learning and subsequently propose a series of contrastive losses that outperform the existing ones…
ETH Zürich（苏黎世联邦理工学院）；Amazon（亚马逊）；MPI-IS（马克斯·普朗克智能系统研究所）
Intensive care units (ICU) are increasingly looking towards machine learning for methods to provide online monitoring of critically ill patients. In machine learning, online monitoring is often formulated as a supervised learning problem. Recently, contrastive learning approaches have demonstrated promising improvements over competitive supervised benchmarks. These methods rely on well-understood data augmentation techniques developed for image data which do not apply to online monitoring. In this work, we overcome this limitation by supplementing time-series data augmentation techniques with …
University of Trento（特伦托大学）；Xiamen University（厦门大学）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）
In this paper, we address Novel Class Discovery (NCD), the task of unveiling new classes in a set of unlabeled samples given a labeled dataset with known classes. We exploit the peculiarities of NCD to build a new framework, named Neighborhood Contrastive Learning (NCL), to learn discriminative representations that are important to clustering performance. Our contribution is twofold. First, we find that a feature extractor trained on the labeled set generates representations in which a generic query sample and its neighbors are likely to share the same class. We exploit this observation to ret…
Sun Yat-sen University（中山大学）；Nanyang Technological University（南洋理工大学）
Graph contrastive learning (GCL), as a popular approach to graph self-supervised learning, has recently achieved a non-negligible effect. To achieve superior performance, the majority of existing GCL methods elaborate on graph data augmentation to construct appropriate contrastive pairs. However, existing methods place more emphasis on the complex graph data augmentation which requires extra time overhead, and pay less attention to developing contrastive schemes specific to encoder characteristics. We argue that a better contrastive scheme should be tailored to the characteristics of graph neu…
Various state-of-the-art self-supervised visual representation learning approaches take advantage of data from multiple sensors by aligning the feature representations across views and/or modalities. In this work, we investigate how aligning representations affects the visual features obtained from cross-view and cross-modal contrastive learning on images and point clouds. On five real-world datasets and on five tasks, we train and evaluate 108 models based on four pretraining variations. We find that cross-modal representation alignment discards complementary visual information, such as color…
Technical University of Munich（慕尼黑工业大学）；Helmholtz Zentrum Munich（慕尼黑亥姆霍兹中心）；Imperial College London（伦敦帝国理工学院）
Image-text contrastive learning has proven effective for pretraining medical image models. When targeting localized downstream tasks like semantic segmentation or object detection, additional local contrastive losses that align image regions with sentences have shown promising results. We study how local contrastive losses are related to global (per-sample) contrastive losses and which effects they have on localized medical downstream tasks. Based on a theoretical comparison, we propose to remove some components of local losses and replace others by a novel distribution prior which enforces un…
Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain specific knowledge. This challenge is magnified in natural language processing where no general rules exist for data augmentation due to the discrete nature of natural language. We tackle this challenge by presenting a Virtual augmentation Supported Contrastive Learning of sentence representations (VaSCL). Originating from the interpretation that data augmentation essentially constructs the neighborhoods of each training instance, we in turn utilize the neighborhood to g…
MIT Computer Science & Artificial Intelligence Lab（麻省理工学院计算机科学与人工智能实验室）
Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, and (2) uniformity of the induced distribution of the (normalized) features on the hypersphere. We prove that, asymptotically, the contrastive loss optimizes these properties, and analyze their positive effects on downstream tasks. Empirically, we introduce an optimizable metric to quantify each property. Extensive experiments on standard vision and language datasets confirm th…
SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome
Wellcome Sanger Institute（沃森桑格研究所）；Cambridge Centre for AI in Medicine（剑桥人工智能医学中心）；Institute of AI for Health（人工智能与健康研究所）；Cambridge Stem Cell Institute（剑桥干细胞研究所）
Graph Soft-Contrastive Learning via Neighborhood Ranking
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；University of Chinese Academy of Sciences（中国科学院大学）；University of Macau（澳门大学）；University of Central Florida（中佛罗里达大学）；Rutgers University（罗格斯大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Multi-level Cross-modal Feature Alignment via Contrastive Learning towards Zero-shot Classification of Remote Sensing Image Scenes
Henan Key Laboratory of Big Data Analysis and Processing（河南省大数据分析与处理重点实验室）；Henan Engineering Laboratory of Spatial Information Processing（河南省空间信息处理工程实验室）；Henan Industrial Technology Academy of Spatio-Temporal Big Data, Henan University（河南大学河南时空大数据产业技术研究院）
Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）
Contrastive learning has been widely used to train transformer-based vision-language models for video-text alignment and multi-modal representation learning. This paper presents a new algorithm called Token-Aware Cascade contrastive learning (TACo) that improves contrastive learning using two novel techniques. The first is the token-aware contrastive loss which is computed by taking into account the syntactic classes of words. This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual co…
Contrastive self-supervised learning (CSL) with a prototypical regularization has been introduced in learning meaningful representations for downstream tasks that require strong semantic information. However, to optimize CSL with a loss that performs the prototypical regularization aggressively, e.g., the ProtoNCE loss, might cause the "coagulation" of examples in the embedding space. That is, the intra-prototype diversity of samples collapses to trivial solutions for their prototype being well-separated from others. Motivated by previous works, we propose to mitigate this phenomenon by learni…
UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels
Department of Electrical & Computer Engineering, The Grainger College of Engineering, UIUC（电气与计算机工程系，格拉inger工程学院，伊利诺伊大学香槟分校）；Siebel School of Computing and Data Science, The Grainger College of Engineering, UIUC（计算与数据科学学院，格拉inger工程学院，伊利诺伊大学香槟分校）；Coordinated Science Laboratory, University of Illinois Urbana-Champaign (UIUC)（协调科学实验室，伊利诺伊大学香槟分校）；VinUni-Illinois Smart Health Center（VinUni-伊利诺伊智能健康中心）
NeuroSpin, CEA Saclay, Université Paris-Saclay（法国原子能和替代能源委员会萨克雷中心 NeuroSpin，巴黎萨克雷大学）；LTCI, Télécom Paris, IPParis（巴黎电信学院 LTCI，巴黎理工学院）
Contrastive Learning has shown impressive results on natural and medical images, without requiring annotated data. However, a particularity of medical images is the availability of meta-data (such as age or sex) that can be exploited for learning representations. Here, we show that the recently proposed contrastive y-Aware InfoNCE loss, that integrates multi-dimensional meta-data, asymptotically optimizes two properties: conditional alignment and global uniformity. Similarly to [Wang, 2020], conditional alignment means that similar samples should have similar features, but conditionally on the…
UiT The Arctic University of Norway（挪威北极大学）；Norwegian Computing Center（挪威计算中心）；University of Copenhagen（哥本哈根大学）
Self-supervised learning is a central component in recent approaches to deep multi-view clustering (MVC). However, we find large variations in the development of self-supervision-based methods for deep MVC, potentially slowing the progress of the field. To address this, we present DeepMVC, a unified framework for deep MVC that includes many recent methods as instances. We leverage our framework to make key observations about the effect of self-supervision, and in particular, drawbacks of aligning representations with contrastive learning. Further, we prove that contrastive alignment can negati…
Nanjing University（南京大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Alibaba Group（阿里巴巴集团）
Graph neural networks (GNNs) have emerged as a powerful paradigm for embedding-based entity alignment due to their capability of identifying isomorphic subgraphs. However, in real knowledge graphs (KGs), the counterpart entities usually have non-isomorphic neighborhood structures, which easily causes GNNs to yield different representations for them. To tackle this problem, we propose a new KG alignment network, namely AliNet, aiming at mitigating the non-isomorphism of neighborhood structures in an end-to-end manner. As the direct neighbors of counterpart entities are usually dissimilar due to…
Multi-modal entity alignment aims to identify equivalent entities between two different multi-modal knowledge graphs, which consist of structural triples and images associated with entities. Most previous works focus on how to utilize and encode information from different modalities, while it is not trivial to leverage multi-modal knowledge in entity alignment because of the modality heterogeneity. In this paper, we propose MCLEA, a Multi-modal Contrastive Learning based Entity Alignment model, to obtain effective joint representations for multi-modal entity alignment. Different from previous …
Perfect Alignment May be Poisonous to Graph Contrastive Learning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Big Data Management and Analysis Methods（北京大数据管理与分析方法重点实验室）
Contrastive Diffusion Alignment: Learning Structured Latents for Controllable Generation
Department of Psychiatry, Weill Cornell Medicine, New York, NY, USA（威立·科林斯医学中心精神科）；Department of Psychiatry, Stanford University, Stanford, CA, USA（斯坦福大学精神科）；Department of Neuroscience, University of Connecticut School of Medicine, Farmington, CT, USA（康涅狄格大学医学院神经科学系）
Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approach
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Defending Against Backdoor Attacks via Alignment Checking in Model-Contrastive Federated Learning
Shandong Computer Science Center（山东计算机科学中心）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；College of computer science and technology, China University of Petroleum（中国石油大学计算机科学与技术学院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Big Data Institute, Qilu University of Technology（齐鲁工业大学大数据研究院）
VaCDA: Variational Contrastive Alignment-based Scalable Human Activity Recognition
Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology（计算机科学与工程系，孟加拉国工程与技术大学）；Department of Computer Science, Bowie State University（计算机科学系，布里沃州立大学）
Monash University（莫纳什大学）；Singapore Management University（新加坡管理大学）；VinUniversity（威宁大学）；The University of Melbourne（墨尔本大学）
Cross-domain graph anomaly detection (CD-GAD) describes the problem of detecting anomalous nodes in an unlabelled target graph using auxiliary, related source graphs with labelled anomalous and normal nodes. Although it presents a promising approach to address the notoriously high false positive issue in anomaly detection, little work has been done in this line of research. There are numerous domain adaptation methods in the literature, but it is difficult to adapt them for GAD due to the unknown distributions of the anomalies and the complex node relations embedded in graph data. To this end,…
Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU
Aalborg University（奥尔堡大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Milestone Systems（里程碑系统公司）；Pioneer Center for Artificial Intelligence（先锋人工智能中心）
MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion
University of Shanghai for Science and TechnologyChina（上海科学技术大学中国）；Shenzhen Institute of Advanced Technology, Chinese Academy of SciencesChina（深圳先进技术研究院，中国科学院中国）；University of Minnesota-Twin Cities, USA（明尼苏达大学双城分校，美国）；University of LeedsUK（利兹大学，英国）
