Data is the key factor to drive the development of machine learning (ML) during the past decade. However, high-quality data, in particular labeled data, is often hard and expensive to collect. To leverage large-scale unlabeled data, self-supervised learning, represented by contrastive learning, is introduced. The objective of contrastive learning is to map different views derived from a training sample (e.g., through data augmentation) closer in their representation space, while different views derived from different samples more distant. In this way, a contrastive model learns to generate informative representations for data samples, which are then used to perform downstream ML tasks. Recent research has shown that machine learning models are vulnerable to various privacy attacks. However, most of the current efforts concentrate on models trained with supervised learning. Meanwhile, data samples' informative representations learned with contrastive learning may cause severe privacy risks as well.
In this paper, we perform the first privacy analysis of contrastive learning through the lens of membership inference and attribute inference. Our experimental results show that contrastive models trained on image datasets are less vulnerable to membership inference attacks but more vulnerable to attribute inference attacks compared to supervised models. The former is due to the fact that contrastive models are less prone to overfitting, while the latter is caused by contrastive models' capability of representing data samples expressively. To remedy this situation, we propose the first privacy-preserving contrastive learning mechanism, Talos, relying on adversarial training. Empirical results show that Talos can successfully mitigate attribute inference risks for contrastive models while maintaining their membership privacy and model utility.
Variational autoencoders (VAEs) suffer from posterior collapse, where the powerful neural networks used for modeling and inference optimize the objective without meaningfully using the latent representation. We introduce inference critics that detect and incentivize against posterior collapse by requiring correspondence between latent variables and the observations. By connecting the critic's objective to the literature in self-supervised contrastive representation learning, we show both theoretically and empirically that optimizing inference critics increases the mutual information between ob…
Graph Contrastive Learning (GCL) has shown promising performance in graph representation learning (GRL) without the supervision of manual annotations. GCL can generate graph-level embeddings by maximizing the Mutual Information (MI) between different augmented views of the same graph (positive pairs). However, the GCL is limited by dimensional collapse, i.e., embedding vectors only occupy a low-dimensional subspace. In this paper, we show that the smoothing effect of the graph pooling and the implicit regularization of the graph convolution are two causes of the dimensional collapse in GCL. To…
Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning
Saw Swee Hock School of Public Health（苏瑞福公共卫生学院）；Institute of Data Science, NUS（新加坡国立大学数据科学研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）
Topology Reorganized Graph Contrastive Learning with Mitigating Semantic Drift
College of Computer Science & Technology, Nanjing University of Aeronautics & Astronautics（南京航空航天大学计算机科学与技术学院）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing University of Aeronautics & Astronautics（南京航空航天大学工信部模式分析与机器智能重点实验室）
Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approach
Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；ByteDance（字节跳动）；National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）
In this paper, we propose a novel training procedure for the continual representation learning problem in which a neural network model is sequentially learned to alleviate catastrophic forgetting in visual search tasks. Our method, called Contrastive Supervised Distillation (CSD), reduces feature forgetting while learning discriminative features. This is achieved by leveraging labels information in a distillation setting in which the student model is contrastively learned from the teacher model. Extensive experiments show that CSD performs favorably in mitigating catastrophic forgetting by out…
FairContrast: Enhancing Fairness through Contrastive learning and Customized Augmenting Methods on Tabular Data
Department of Industrial Engineering University of Central Florida（工业工程系，中央佛罗里达大学）；Department of Computer Science University of Central Florida（计算机科学系，中央佛罗里达大学）
Google（谷歌）；Google Research（谷歌研究院）；TTI-Chicago（芝加哥丰田技术研究所）；University of Chicago（芝加哥大学）
Self-supervised representation learning has made significant leaps fueled by progress in contrastive learning, which seeks to learn transformations that embed positive input pairs nearby, while pushing negative pairs far apart. While positive pairs can be generated reliably (e.g., as different views of the same image), it is difficult to accurately establish negative pairs, defined as samples from different images regardless of their semantic content or visual features. A fundamental problem in contrastive learning is mitigating the effects of false negatives. Contrasting false negatives induc…
Mask-informed Deep Contrastive Incomplete Multi-view Clustering
Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；School of Computer Science, China University of Geosciences（中国地质大学计算机学院）
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Qwen Large Model Application Team, Alibaba Institute of Automation, Chinese Academy of Sciences（文勤大模型应用团队，阿里巴巴自动化研究所，中国科学院）；Qwen Large Model Application Team, Alibaba（文勤大模型应用团队，阿里巴巴）
DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series
Department of Data Science and AI, Monash University（数据科学与人工智能系，墨尔本大学）；Department of Computer Science, University of Oxford（计算机科学系，牛津大学）；IBM T. J. Watson Research Center（IBM T. J. Watson研究中心）；Squirrel Ai Learning（Squirrel Ai Learning公司）；School of ICT, Griffith University（信息科技学院，格里菲斯大学）
Deep learning has been a prevalence in computational chemistry and widely implemented in molecule property predictions. Recently, self-supervised learning (SSL), especially contrastive learning (CL), gathers growing attention for the potential to learn molecular representations that generalize to the gigantic chemical space. Unlike supervised learning, SSL can directly leverage large unlabeled data, which greatly reduces the effort to acquire molecular property labels through costly and time-consuming simulations or experiments. However, most molecular SSL methods borrow the insights from the …
This paper presents simple and efficient methods to mitigate sampling bias in active learning while achieving state-of-the-art accuracy and model robustness. We introduce supervised contrastive active learning by leveraging the contrastive loss for active learning under a supervised setting. We propose an unbiased query strategy that selects informative data samples of diverse feature representations with our methods: supervised contrastive active learning (SCAL) and deep feature modeling (DFM). We empirically demonstrate our proposed methods reduce sampling bias, achieve state-of-the-art accu…
Quantifying and Mitigating Self-Preference Bias of LLM Judges
CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China（复杂实验室、计算机科学与工程学院、电子科技大学、成都、中国）
EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning
School of Computing, Wichita State University（威奇托州立大学计算学院）；Department of Computer Science, American International University-Bangladesh（美国国际孟加拉大学计算机科学系）
Analyzing and Mitigating Poisoning Attacks on Flow-Based Intrusion Detection
CortAIx Labs, Thales Deutschland（Thales Deutschland）；Grey Solutions；University of Osnabrück（奥斯纳布吕克大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；Ulm University（乌尔姆大学）
A significant challenge in reinforcement learning is quantifying the complex relationship between actions and long-term rewards. The effects may manifest themselves over a long sequence of state-action pairs, making them hard to pinpoint. In this paper, we propose a method to link transitions with significant deviations in state with unusually large variations in subsequent rewards. Such transitions are marked as possible causal effects, and the corresponding state-action pairs are added to a separate replay buffer. In addition, we include \textit{contrastive} samples corresponding to transiti…
Brandeis University（布兰迪斯大学）；University of Virginia（弗吉尼亚大学）；University of Notre Dame（圣母大学）
Prevailing deep graph learning models often suffer from label sparsity issue. Although many graph few-shot learning (GFL) methods have been developed to avoid performance degradation in face of limited annotated data, they excessively rely on labeled data, where the distribution shift in the test phase might result in impaired generalization ability. Additionally, they lack a general purpose as their designs are coupled with task or data-specific characteristics. To this end, we propose a general and effective Contrastive Graph Few-shot Learning framework (CGFL). CGFL leverages a self-distille…
VaCDA: Variational Contrastive Alignment-based Scalable Human Activity Recognition
Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology（计算机科学与工程系，孟加拉国工程与技术大学）；Department of Computer Science, Bowie State University（计算机科学系，布里沃州立大学）
FedCoSR: Personalized Federated Learning with Contrastive Shareable Representations for Label Heterogeneity in Non-IID Data
Department of Data Science and AI, Faculty of IT and Monash Energy Institute, Monash University（数据科学与人工智能系，IT学院和莫纳什能源研究所，莫纳什大学）；School of Computer Science and Technology, University of Electronic Science and Technology of China (UESTC), Chengdu, and Shenzhen Institute for Advanced Study of UESTC, Shenzhen, China（计算机科学与技术学院，电子科技大学（UESTC）成都校区，以及UESTC深圳高级研究所，深圳，中国）
