Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data
MPI for Informatics（马克斯·普朗克信息学研究所）；Saarland Informatics Campus（萨尔兰信息学园区）；Goethe University Frankfurt（法兰克福歌德大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；University of Oxford（牛津大学）
Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, Christian Rupprecht
Most approaches for self-supervised learning (SSL) are optimised on curated balanced datasets, e.g. ImageNet, despite the fact that natural data usually exhibits long-tail distributions. In this paper, we analyse the behaviour of one of the most popular variants of SSL, i.e. contrastive methods, on long-tail data. In particular, we investigate the role of the temperature parameter $τ$ in the contrastive loss, by analysing the loss through the lens of average distance maximisation, and find that a large $τ$ emphasises group-wise discrimination, whereas a small $τ$ leads to a higher degree of instance discrimination. While $τ$ has thus far been treated exclusively as a constant hyperparameter, in this work, we propose to employ a dynamic $τ$ and show that a simple cosine schedule can yield significant improvements in the learnt representations. Such a schedule results in a constant `task switching' between an emphasis on instance discrimination and group-wise discrimination and thereby ensures that the model learns both group-wise features, as well as instance-specific details. Since frequent classes benefit from the former, while infrequent classes require the latter, we find this method to consistently improve separation between the classes in long-tail data without any additional computational cost.
Fudan University（复旦大学）；Shanghai Collaborative Innovation Center on Intelligent Visual Computing（上海智能视觉计算协同创新中心）；La Trobe University（乐卓博大学）
Real-world data typically follow a long-tailed distribution, where a few majority categories occupy most of the data while most minority categories contain a limited number of samples. Classification models minimizing cross-entropy struggle to represent and classify the tail classes. Although the problem of learning unbiased classifiers has been well studied, methods for representing imbalanced data are under-explored. In this paper, we focus on representation learning for imbalanced data. Recently, supervised contrastive learning has shown promising performance on balanced data recently. Howe…
University of Pisa（比萨大学）；TU Graz（格拉茨工业大学）；CSH Vienna（维也纳复杂科学中心）
We study the impact of different pruning techniques on the representation learned by deep neural networks trained with contrastive loss functions. Our work finds that at high sparsity levels, contrastive learning results in a higher number of misclassified examples relative to models trained with traditional cross-entropy loss. To understand this pronounced difference, we use metrics such as the number of PIEs (Hooker et al., 2019), Q-Score (Kalibhat et al., 2022), and PD-Score (Baldock et al., 2021) to measure the impact of pruning on the learned representation quality. Our analysis suggests …
Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Xiaomi Inc.（小米公司）；School of Software Technology, Zhejiang University（浙江大学软件学院）
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Hong Kong Baptist University（香港浸会大学）
Self-supervised learning has achieved a great success in the representation learning of visual and textual data. However, the current methods are mainly validated on the well-curated datasets, which do not exhibit the real-world long-tailed distribution. Recent attempts to consider self-supervised long-tailed learning are made by rebalancing in the loss perspective or the model perspective, resembling the paradigms in the supervised long-tailed learning. Nevertheless, without the aid of labels, these explorations have not shown the expected significant promise due to the limitation in tail sam…
Unsupervised Contrastive Learning Using Out-Of-Distribution Data for Long-Tailed Dataset
Department of Electronic Engineering, Seoul National University of Science and Technology（电子工程系，首尔科学技术大学）；Department of Electrical and Information Engineering, Seoul National University of Science and Technology（电气与信息工程系，首尔科学技术大学）；School of Electrical and Electronics Engineering, Chung-Ang University（电气电子工程学院， Chung-Ang 大学）
Angular Gaussian Supervised Contrastive Learning for Long-Tailed Electrocardiogram Arrhythmia Diagnosis
School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Department of Neurology and Neurological Rehabilitation, Shanghai Yangzhi Rehabilitation Hospital, School of Medicine, Tongji University（同济大学医学院附属上海养志康复医院神经科与神经康复科）
Contrastive learning is among the most successful methods for visual representation learning, and its performance can be further improved by jointly performing clustering on the learned representations. However, existing methods for joint clustering and contrastive learning do not perform well on long-tailed data distributions, as majority classes overwhelm and distort the loss of minority classes, thus preventing meaningful representations to be learned. Motivated by this, we develop a novel joint clustering and contrastive learning framework by adapting the debiased contrastive loss to avoid…
Contrastive self-supervised learning has outperformed supervised pretraining on many downstream tasks like segmentation and object detection. However, current methods are still primarily applied to curated datasets like ImageNet. In this paper, we first study how biases in the dataset affect existing methods. Our results show that current contrastive approaches work surprisingly well across: (i) object- versus scene-centric, (ii) uniform versus long-tailed and (iii) general versus domain-specific datasets. Second, given the generality of the approach, we try to realize further gains with minor…
Texas A&M University（德州农工大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
The recent breakthrough achieved by contrastive learning accelerates the pace for deploying unsupervised training on real-world data applications. However, unlabeled data in reality is commonly imbalanced and shows a long-tail distribution, and it is unclear how robustly the latest contrastive learning methods could perform in the practical scenario. This paper proposes to explicitly tackle this challenge, via a principled framework called Self-Damaging Contrastive Learning (SDCLR), to automatically balance the representation learning without knowing the classes. Our main inspiration is drawn …
Recent works have shown that self-supervised learning can achieve remarkable robustness when integrated with adversarial training (AT). However, the robustness gap between supervised AT (sup-AT) and self-supervised AT (self-AT) remains significant. Motivated by this observation, we revisit existing self-AT methods and discover an inherent dilemma that affects self-AT robustness: either strong or weak data augmentations are harmful to self-AT, and a medium strength is insufficient to bridge the gap. To resolve this dilemma, we propose a simple remedy named DYNACL (Dynamic Adversarial Contrastiv…
Missouri University of Science and Technology（密苏里科技大学）
Instance-level Image Retrieval (IIR), or simply Instance Retrieval, deals with the problem of finding all the images within an dataset that contain a query instance (e.g. an object). This paper makes the first attempt that tackles this problem using instance-discrimination based contrastive learning (CL). While CL has shown impressive performance for many computer vision tasks, the similar success has never been found in the field of IIR. In this work, we approach this problem by exploring the capability of deriving discriminative representations from pre-trained and fine-tuned CL models. To b…
Texas A&M University（得克萨斯农工大学）；University of Texas at Austin（得克萨斯大学奥斯汀分校）；Google Research, Brain Team（谷歌研究院大脑团队）
Contrastive learning approaches have achieved great success in learning visual representations with few labels of the target classes. That implies a tantalizing possibility of scaling them up beyond a curated "seed" benchmark, to incorporating more unlabeled images from the internet-scale external sources to enhance its performance. However, in practice, larger amount of unlabeled data will require more computing resources due to the bigger model size and longer training needed. Moreover, open-world unlabeled data usually follows an implicit long-tail class or attribute distribution, many of w…
The Hong Kong Polytechnic University（香港理工大学）；University of California Irvine（加州大学欧文分校）
Knowledge Graph (KG) errors introduce non-negligible noise, severely affecting KG-related downstream tasks. Detecting errors in KGs is challenging since the patterns of errors are unknown and diverse, while ground-truth labels are rare or even unavailable. A traditional solution is to construct logical rules to verify triples, but it is not generalizable since different KGs have distinct rules with domain knowledge involved. Recent studies focus on designing tailored detectors or ranking triples based on KG embedding loss. However, they all rely on negative samples for training, which are gene…
Relation prediction is a task designed for knowledge graph completion which aims to predict missing relationships between entities. Recent subgraph-based models for inductive relation prediction have received increasing attention, which can predict relation for unseen entities based on the extracted subgraph surrounding the candidate triplet. However, they are not completely inductive because of their disability of predicting unseen relations. Moreover, they fail to pay sufficient attention to the role of relation as they only depend on the model to learn parameterized relation embedding, whic…
A Modality-Tailored Graph Modeling Framework for Urban Region Representation via Contrastive Learning
Center for Applied Statistics, School of Statistics, Innovation Platform, Renmin University of China（应用统计中心、统计学院、创新平台、中国人民大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳学院）
Contrastive learning pre-trains an image encoder using a large amount of unlabeled data such that the image encoder can be used as a general-purpose feature extractor for various downstream tasks. In this work, we propose PoisonedEncoder, a data poisoning attack to contrastive learning. In particular, an attacker injects carefully crafted poisoning inputs into the unlabeled pre-training data, such that the downstream classifiers built based on the poisoned encoder for multiple target downstream tasks simultaneously classify attacker-chosen, arbitrary clean inputs as attacker-chosen, arbitrary …
Sun Yat-sen University（中山大学）；Nanyang Technological University（南洋理工大学）
Graph contrastive learning (GCL), as a popular approach to graph self-supervised learning, has recently achieved a non-negligible effect. To achieve superior performance, the majority of existing GCL methods elaborate on graph data augmentation to construct appropriate contrastive pairs. However, existing methods place more emphasis on the complex graph data augmentation which requires extra time overhead, and pay less attention to developing contrastive schemes specific to encoder characteristics. We argue that a better contrastive scheme should be tailored to the characteristics of graph neu…
Beijing University of Posts and Telecommunications（北京邮电大学）；BioMap（百图生科）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
Recent studies show that graph convolutional network (GCN) often performs worse for low-degree nodes, exhibiting the so-called structural unfairness for graphs with long-tailed degree distributions prevalent in the real world. Graph contrastive learning (GCL), which marries the power of GCN and contrastive learning, has emerged as a promising self-supervised approach for learning node representations. How does GCL behave in terms of structural fairness? Surprisingly, we find that representations obtained by GCL methods are already fairer to degree bias than those learned by GCN. We theoretical…
AdNGCL: Adaptive Negative Scheduling for Graph Contrastive Learning
Jinan University（暨南大学）；University of Science and Technology of China（中国科学技术大学）；Hainan University（海南大学）；Manchester Metropolitan University（曼彻斯特城市大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）
DICE: Device-level Integrated Circuits Encoder with Graph Contrastive Pretraining
Electrical and Computer Engineering, The University of Texas at Austin（得克萨斯大学奥斯汀分校电子与计算机工程系）；Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Computer Science, The University of Hong Kong（香港大学计算机科学系）
Center for Research on Intelligent Perception and Computing（智能感知与计算研究中心）；National Laboratory of Pattern Recognition（模式识别国家实验室）；Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）
Corporate credit rating reflects the level of corporate credit and plays a crucial role in modern financial risk control. But real-world credit rating data usually shows long-tail distributions, which means heavy class imbalanced problem challenging the corporate credit rating system greatly. To tackle that, inspried by the recent advances of pre-train techniques in self-supervised representation learning, we propose a novel framework named Contrastive Pre-training for Corporate Credit Rating (CP4CCR), which utilizes the self-surpervision for getting over class imbalance. Specifically, we prop…
MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data
University of Bonn（波恩大学）；MPI for Informatics, SIC（信息研究所）；Tuebingen AI Center/University of Tuebingen（图宾根人工智能中心/图宾根大学）；MIT-IBM Watson AI Lab（麻省理工-IBM Watson人工智能实验室）
