Confidence-aware Contrastive Learning for Selective Classification
National Key Laboratory for Novel Software Technology（计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；College of Computer Science and Software Engineering, Hohai University（河海大学计算机与软件学院）；Key Laboratory of Water Big Data Technology of Ministry of Water Resources（水利部水大数据技术重点实验室）
Selective classification enables models to make predictions only when they are sufficiently confident, aiming to enhance safety and reliability, which is important in high-stakes scenarios. Previous methods mainly use deep neural networks and focus on modifying the architecture of classification layers to enable the model to estimate the confidence of its prediction. This work provides a generalization bound for selective classification, disclosing that optimizing feature layers helps improve the performance of selective classification. Inspired by this theory, we propose to explicitly improve the selective classification model at the feature level for the first time, leading to a novel Confidence-aware Contrastive Learning method for Selective Classification, CCL-SC, which similarizes the features of homogeneous instances and differentiates the features of heterogeneous instances, with the strength controlled by the model's confidence. The experimental results on typical datasets, i.e., CIFAR-10, CIFAR-100, CelebA, and ImageNet, show that CCL-SC achieves significantly lower selective risk than state-of-the-art methods, across almost all coverage degrees. Moreover, it can be combined with existing methods to bring further improvement.
University of Pittsburgh（匹兹堡大学）；University of Notre Dame（圣母大学）
After a model is deployed on edge devices, it is desirable for these devices to learn from unlabeled data to continuously improve accuracy. Contrastive learning has demonstrated its great potential in learning from unlabeled data. However, the online input data are usually none independent and identically distributed (non-iid) and storages of edge devices are usually too limited to store enough representative data from different data classes. We propose a framework to automatically select the most representative data from the unlabeled input stream, which only requires a small data buffer for …
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；The University of Sydney（悉尼大学）
Deep networks have strong capacities of embedding data into latent representations and finishing following tasks. However, the capacities largely come from high-quality annotated labels, which are expensive to collect. Noisy labels are more affordable, but result in corrupted representations, leading to poor generalization performance. To learn robust representations and handle noisy labels, we propose selective-supervised contrastive learning (Sel-CL) in this paper. Specifically, Sel-CL extend supervised contrastive learning (Sup-CL), which is powerful in representation learning, but is degra…
Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction
School of Intelligence Science and Technology, Peking University, Beijing（智能科学与技术学院，北京大学，北京）；Computational Intelligence Laboratory（计算智能实验室）；Institute for Artificial Intelligence（人工智能研究院）；State Key Laboratory of General Artificial Intelligence, Peking University, Beijing（通用人工智能国家重点实验室，北京大学，北京）
Breaking the Prototype Bias Loop: Confidence-Aware Federated Contrastive Learning for Highly Imbalanced Clients
Key Laboratory of Water Big Data Technology of Ministry of Water Resources, College of Computer Science and Software Engineering, Hohai University, Nanjing, China（水利部水大数据技术重点实验室，计算机科学与软件工程学院，河海大学，南京，中国）；Department of Computer Science, City University of Hong Kong, Hong Kong, China（计算机科学系，香港城市大学，香港，中国）；State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China（新型软件技术国家重点实验室，南京大学，南京，中国）；School of Artificial Intelligence and Information Technology, Nanjing University of Chinese Medicine, Nanjing, China（人工智能与信息技术学院，南京中医药大学，南京，中国）
Learning discriminative representations of unlabelled data is a challenging task. Contrastive self-supervised learning provides a framework to learn meaningful representations using learned notions of similarity measures from simple pretext tasks. In this work, we propose a simple and efficient framework for self-supervised image segmentation using contrastive learning on image patches, without using explicit pretext tasks or any further labeled fine-tuning. A fully convolutional neural network (FCNN) is trained in a self-supervised manner to discern features in the input images and obtain con…
Contrastive approaches to representation learning have recently shown great promise. In contrast to generative approaches, these contrastive models learn a deterministic encoder with no notion of uncertainty or confidence. In this paper, we introduce a simple approach based on "contrasting distributions" that learns to assign uncertainty for pretrained contrastive representations. In particular, we train a deep network from a representation to a distribution in representation space, whose variance can be used as a measure of confidence. In our experiments, we show that this deep uncertainty mo…
National University of Defense Technology（国防科技大学）；University of Science and Technology of China（中国科学技术大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Benefiting from the intrinsic supervision information exploitation capability, contrastive learning has achieved promising performance in the field of deep graph clustering recently. However, we observe that two drawbacks of the positive and negative sample construction mechanisms limit the performance of existing algorithms from further improvement. 1) The quality of positive samples heavily depends on the carefully designed data augmentations, while inappropriate data augmentations would easily lead to the semantic drift and indiscriminative positive samples. 2) The constructed negative samp…
National University of Singapore（新加坡国立大学）；Youtu Lab Tencent（腾讯优图实验室）
This paper proposes a probabilistic contrastive loss function for self-supervised learning. The well-known contrastive loss is deterministic and involves a temperature hyperparameter that scales the inner product between two normed feature embeddings. By reinterpreting the temperature hyperparameter as a quantity related to the radius of the hypersphere, we derive a new loss function that involves a confidence measure which quantifies uncertainty in a mathematically grounding manner. Some intriguing properties of the proposed loss function are empirically demonstrated, which agree with human-l…
Achieving artificially intelligent-native wireless networks is necessary for the operation of future 6G applications such as the metaverse. Nonetheless, current communication schemes are, at heart, a mere reconstruction process that lacks reasoning. One key solution that enables evolving wireless communication to a human-like conversation is semantic communications. In this paper, a novel machine reasoning framework is proposed to pre-process and disentangle source data so as to make it semantic-ready. In particular, a novel contrastive learning framework is proposed, whereby instance and clus…
Probabilistic Contrastive Learning with Explicit Concentration on the Hypersphere
Athinoula A. Martinos Center（阿西努拉·A.马蒂诺斯中心）；MGH, HMS, Harvard University（哈佛医学院、麻省总医院、哈佛大学）；Imperial College London（伦敦帝国学院）；Technical University of Munich（慕尼黑工业大学）；University of Zurich（苏黎世大学）
Contrastive learning has demonstrated great capability to learn representations without annotations, even outperforming supervised baselines. However, it still lacks important properties useful for real-world application, one of which is uncertainty. In this paper, we propose a simple way to generate uncertainty scores for many contrastive methods by re-purposing temperature, a mysterious hyperparameter used for scaling. By observing that temperature controls how sensitive the objective is to specific embedding locations, we aim to learn temperature as an input-dependent variable, treating it …
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data. Here, the choice of data augmentation is sensitive to the quality of learned representations: as harder the data augmentations are applied, the views share more task-relevant information, but also task-irrelevant one that can hinder the generalization capability of representation. Motivated by this, we present a new robust contrastive learning scheme, coined RényiCL, which can effectively manage harder augmentations by utilizing Rényi divergence. Our …
This paper proposes to perform online clustering by conducting twin contrastive learning (TCL) at the instance and cluster level. Specifically, we find that when the data is projected into a feature space with a dimensionality of the target cluster number, the rows and columns of its feature matrix correspond to the instance and cluster representation, respectively. Based on the observation, for a given dataset, the proposed TCL first constructs positive and negative pairs through data augmentations. Thereafter, in the row and column space of the feature matrix, instance- and cluster-level con…
Learning Robust Representations for Malicious Content Detection via Contrastive Sampling and Uncertainty Estimation
1 College of Engineering；Computer Science, University of Central Florida, Orlando, USA, 2 Department of Computer Science \& Engineering, Mississippi State University, MS State, USA , 3 Department of Computer Science, University of Nevada, Las Vegas, USA
UC Berkeley（加州大学伯克利分校）；UT Austin（德克萨斯大学奥斯汀分校）；Google Brain（谷歌大脑）；University of Alberta（阿尔伯塔大学）
It is common to address the curse of dimensionality in Markov decision processes (MDPs) by exploiting low-rank representations. This motivates much of the recent theoretical study on linear MDPs. However, most approaches require a given representation under unrealistic assumptions about the normalization of the decomposition or introduce unresolved computational challenges in practice. Instead, we consider an alternative definition of linear MDPs that automatically ensures normalization while allowing efficient representation learning via contrastive estimation. The framework also admits confi…
We propose a novel semi-supervised learning (SSL) method that adopts selective training with pseudo labels. In our method, we generate hard pseudo-labels and also estimate their confidence, which represents how likely each pseudo-label is to be correct. Then, we explicitly select which pseudo-labeled data should be used to update the model. Specifically, assuming that loss on incorrectly pseudo-labeled data sensitively increase against data augmentation, we select the data corresponding to relatively small loss after applying data augmentation. The confidence is used not only for screening can…
Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inference
King’s Communications, Learning & Information Processing (KCLIP) lab（国王通讯、学习与信息处理（KCLIP）实验室）；Centre for Intelligent Information Processing Systems (CIIPS)（智能信息处理系统中心）；Department of Engineering, King’s College London（伦敦国王学院工程系）
School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
University of Central Florida（中佛罗里达大学）；Google Research（谷歌研究院）
We study the problem of semantic segmentation calibration. Lots of solutions have been proposed to approach model miscalibration of confidence in image classification. However, to date, confidence calibration research on semantic segmentation is still limited. We provide a systematic study on the calibration of semantic segmentation models and propose a simple yet effective approach. First, we find that model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. Among them, prediction correctness, especially misprediction, is more important to miscali…
University of Southern California（南加州大学）；Universität für Musik und darstellende Kunst Graz（格拉茨音乐与表演艺术大学）
Deep learning methods have shown suitability for time series classification in the health and medical domain, with promising results for electrocardiogram data classification. Successful identification of myocardial infarction holds life saving potential and any meaningful improvement upon deep learning models in this area is of great interest. Conventionally, data augmentation methods are applied universally to the training set when data are limited in order to ameliorate data resolution or sample size. In the method proposed in this study, data augmentation was not applied in the context of …
South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Alibaba Group（阿里巴巴集团）
Despite the success of deep neural network (DNN) on sequential data (i.e., scene text and speech) recognition, it suffers from the over-confidence problem mainly due to overfitting in training with the cross-entropy loss, which may make the decision-making less reliable. Confidence calibration has been recently proposed as one effective solution to this problem. Nevertheless, the majority of existing confidence calibration methods aims at non-sequential data, which is limited if directly applied to sequential data since the intrinsic contextual dependency in sequences or the class-specific sta…
Selective Learning: Towards Robust Calibration with Dynamic Regularization
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Rutgers University（罗格斯大学）；A*STAR Centre for Frontier AI Research (CFAR)（新加坡科技研究局前沿人工智能研究中心）
Overcoming Common Flaws in the Evaluation of Selective Classification Systems
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Imaging（亥姆霍兹成像机构）；University of Heidelberg（海德堡大学）；Heidelberg University Hospital（海德堡大学医院）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）
Stanford University（斯坦福大学）；Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）
We propose a selective learning method using meta-learning and deep reinforcement learning for medical image interpretation in the setting of limited labeling resources. Our method, MedSelect, consists of a trainable deep learning selector that uses image embeddings obtained from contrastive pretraining for determining which images to label, and a non-parametric selector that uses cosine similarity to classify unseen images. We demonstrate that MedSelect learns an effective selection strategy outperforming baseline selection strategies across seen and unseen medical conditions for chest X-ray …
Selective Ensemble Based on Preference-Directed Multi-Objective Bandits
Department of Computer Science, Zhejiang Gongshang University（浙江工商大学计算机科学与技术学院）；Center for Advanced Intelligence Project, RIKEN（日本理化学研究所先进智能研究中心）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学新领域创成科学研究科）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
Technion - Israel Institute of Technology（以色列理工学院）
We consider the problem of selective prediction (also known as reject option) in deep neural networks, and introduce SelectiveNet, a deep neural architecture with an integrated reject option. Existing rejection mechanisms are based mostly on a threshold over the prediction confidence of a pre-trained network. In contrast, SelectiveNet is trained to optimize both classification (or regression) and rejection simultaneously, end-to-end. The result is a deep neural network that is optimized over the covered domain. In our experiments, we show a consistently improved risk-coverage trade-off over se…
Enhancing Continual Learning for Software Vulnerability Prediction: Addressing Catastrophic Forgetting via Hybrid-Confidence-Aware Selective Replay for Temporal LLM Fine-Tuning
School of Computer Science, University of Nottingham, Nottingham, United Kingdom（诺丁汉大学计算机科学学院）；School of Informatics and Computing, Northern Arizona University, Flagstaff, United States of America（北亚利桑那大学信息学院）；School of Computer Science, University of Nottingham, Ningbo, China（诺丁汉大学宁波校区）
Recent work has shown that models trained to the same objective, and which achieve similar measures of accuracy on consistent test data, may nonetheless behave very differently on individual predictions. This inconsistency is undesirable in high-stakes contexts, such as medical diagnosis and finance. We show that this inconsistent behavior extends beyond predictions to feature attributions, which may likewise have negative implications for the intelligibility of a model, and one's ability to find recourse for subjects. We then introduce selective ensembles to mitigate such inconsistencies by a…
