Learning good image representations that are beneficial to downstream tasks is a challenging task in computer vision. As such, a wide variety of self-supervised learning approaches have been proposed. Among them, contrastive learning has shown competitive performance on several benchmark datasets. The embeddings of contrastive learning are arranged on a hypersphere that results in using the inner (dot) product as a distance measurement in Euclidean space. However, the underlying structure of many scientific fields like social networks, brain imaging, and computer graphics data exhibit highly non-Euclidean latent geometry. We propose a novel contrastive learning framework to learn semantic relationships in the hyperbolic space. Hyperbolic space is a continuous version of trees that naturally owns the ability to model hierarchical structures and is thus beneficial for efficient contrastive representation learning. We also extend the proposed Hyperbolic Contrastive Learning (HCL) to the supervised domain and studied the adversarial robustness of HCL. The comprehensive experiments show that our proposed method achieves better results on self-supervised pretraining, supervised classification, and higher robust accuracy than baseline methods.
University of Maryland, College Park（马里兰大学帕克分校）；Google Research（谷歌研究院）；Meta
Although self-/un-supervised methods have led to rapid progress in visual representation learning, these methods generally treat objects and scenes using the same lens. In this paper, we focus on learning representations for objects and scenes that preserve the structure among them. Motivated by the observation that visually similar objects are close in the representation space, we argue that the scenes and objects should instead follow a hierarchical structure based on their compositionality. To exploit such a structure, we propose a contrastive learning framework where a Euclidean loss is us…
Harbin Institute of Technology(Shenzhen)（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）；Huawei Technologies co.ltd（华为技术有限公司）；Shenzhen University（深圳大学）
Recently, hyperbolic space has risen as a promising alternative for semi-supervised graph representation learning. Many efforts have been made to design hyperbolic versions of neural network operations. However, the inspiring geometric properties of this unique geometry have not been fully explored yet. The potency of graph models powered by the hyperbolic space is still largely underestimated. Besides, the rich information carried by abundant unlabelled samples is also not well utilized. Inspired by the recently active and emerging self-supervised learning, in this study, we attempt to enhanc…
Data Augmentation of Contrastive Learning is Estimating Positive-incentive Noise
The University of Hong Kong（香港大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院）；Fudan University（复旦大学）；School of Artificial Intelligence, OPtics（人工智能学院）；ElectroNics (iOPEN), Northwestern Polytechnical University（西北工业大学电子学院）
Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU
Aalborg University（奥尔堡大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Milestone Systems（里程碑系统公司）；Pioneer Center for Artificial Intelligence（先锋人工智能中心）
There have been several recent efforts towards developing representations for multivariate time-series in an unsupervised learning framework. Such representations can prove beneficial in tasks such as activity recognition, health monitoring, and anomaly detection. In this paper, we consider a setting where we observe time-series at each node in a dynamic graph. We propose a framework called GraphTNC for unsupervised learning of joint representations of the graph and the time-series. Our approach employs a contrastive learning strategy. Based on an assumption that the time-series and graph evol…
Amazon Web Services（亚马逊云科技）；UMass Amherst（马萨诸塞大学阿默斯特分校）；UCLA（加州大学洛杉矶分校）
Few-shot learning aims to transfer information from one task to enable generalization on novel tasks given a few examples. This information is present both in the domain and the class labels. In this work we investigate the complementary roles of these two sources of information by combining instance-discriminative contrastive learning and supervised learning in a single framework called Supervised Momentum Contrastive learning (SUPMOCO). Our approach avoids a problem observed in supervised learning where information in images not relevant to the task is discarded, which hampers their generali…
A recent popular approach to out-of-distribution (OOD) detection is based on a self-supervised learning technique referred to as contrastive learning. There are two main variants of contrastive learning, namely instance and class discrimination, targeting features that can discriminate between different instances for the former, and different classes for the latter. In this paper, we aim to understand the effectiveness and limitation of existing contrastive learning methods for OOD detection. We approach this in 3 ways. First, we systematically study the performance difference between the inst…
University of Trento（特伦托大学）；Skolkovo Institute of Science and Technology（斯科尔科沃科技学院）；Skoltech（斯科尔科沃科技学院）；Yandex（杨德克斯）；AIRI（人工智能研究所）
Metric learning aims to learn a highly discriminative model encouraging the embeddings of similar classes to be close in the chosen metrics and pushed apart for dissimilar ones. The common recipe is to use an encoder to extract embeddings and a distance-based loss function to match the representations -- usually, the Euclidean distance is utilized. An emerging interest in learning hyperbolic data embeddings suggests that hyperbolic geometry can be beneficial for natural data. Following this line of work, we propose a new hyperbolic-based model for metric learning. At the core of our method is …
The state-of-the-art unsupervised contrastive visual representation learning methods that have emerged recently (SimCLR, MoCo, SwAV) all make use of data augmentations in order to construct a pretext task of instant discrimination consisting of similar and dissimilar pairs of images. Similar pairs are constructed by randomly extracting patches from the same image and applying several other transformations such as color jittering or blurring, while transformed patches from different image instances in a given batch are regarded as dissimilar pairs. We argue that this approach can result similar…
University of Rome Tor Vergata（罗马第二大学）；University of Cambridge（剑桥大学）；Athinoula A. Martinos Center for Biomedical Imaging（阿西诺拉·A·马蒂诺斯生物医学成像中心）
The lack of large labeled medical imaging datasets, along with significant inter-individual variability compared to clinically established disease classes, poses significant challenges in exploiting medical imaging information in a precision medicine paradigm, where in principle dense patient-specific data can be employed to formulate individual predictions and/or stratify patients into finer-grained groups which may follow more homogeneous trajectories and therefore empower clinical trials. In order to efficiently explore the effective degrees of freedom underlying variability in medical imag…
ASIL: Augmented Structural Information Learning for Deep Graph Clustering in Hyperbolic Space
Beijing University of Posts and Telecommunications（北京邮电大学）；North China Electric Power University（华北电力大学）；DiDi Chuxing（滴滴出行）；Beihang University（北航）；University of Illinois at Chicago（伊利诺伊大学香槟分校）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Baidu Inc.（百度公司）；Griffith University（格里菲斯大学）；Xiaomi AI Lab（小米人工智能实验室）
Self-supervised learning (especially contrastive learning) methods on heterogeneous graphs can effectively get rid of the dependence on supervisory data. Meanwhile, most existing representation learning methods embed the heterogeneous graphs into a single geometric space, either Euclidean or hyperbolic. This kind of single geometric view is usually not enough to observe the complete picture of heterogeneous graphs due to their rich semantics and complex structures. Under these observations, this paper proposes a novel self-supervised learning method, termed as Geometry Contrastive Learning (GC…
We propose methods to strengthen the invariance properties of representations obtained by contrastive learning. While existing approaches implicitly induce a degree of invariance as representations are learned, we look to more directly enforce invariance in the encoding process. To this end, we first introduce a training objective for contrastive learning that uses a novel regularizer to control how the representation changes under transformation. We show that representations trained with this objective perform better on downstream tasks and are more robust to the introduction of nuisance tran…
University of Cambridge（剑桥大学）；MRC Epidemiology Unit（医学研究委员会流行病学组）；Alan Turing Institute（艾伦·图灵研究所）
Human Activity Recognition (HAR) constitutes one of the most important tasks for wearable and mobile sensing given its implications in human well-being and health monitoring. Motivated by the limitations of labeled datasets in HAR, particularly when employed in healthcare-related applications, this work explores the adoption and adaptation of SimCLR, a contrastive learning technique for visual representations, to HAR. The use of contrastive learning objectives causes the representations of corresponding views to be more similar, and those of non-corresponding views to be more different. After …
Hyperbolic Multimodal Representation Learning for Biological Taxonomies
Simon Fraser University（西蒙弗雷泽大学）；University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）；University of Guelph（圭尔夫大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）；Aalborg University（奥胡斯大学）
Hyperbolic Enhanced Representation Learning for Incomplete Multi-view Clustering
School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Beijing Key Laboratory of Traffic Data Analysis and Mining, Beijing Jiaotong University（北京交通大学交通数据挖掘与分析重点实验室）；School of Medicine, Zhejiang University（浙江大学医学院）
Bootstrap Your Own Latent (BYOL) introduced an approach to self-supervised learning avoiding the contrastive paradigm and subsequently removing the computational burden of negative sampling associated with such methods. However, we empirically find that the image representations produced under the BYOL's self-distillation paradigm are poorly distributed in representation space compared to contrastive methods. This work empirically demonstrates that feature diversity enforced by contrastive losses is beneficial to image representation uniformity when employed in BYOL, and as such, provides grea…
Zhejiang University（浙江大学）；Chongqing University（重庆大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
Partial label learning (PLL) is an important problem that allows each training example to be labeled with a coarse candidate set, which well suits many real-world data annotation scenarios with label ambiguity. Despite the promise, the performance of PLL often lags behind the supervised counterpart. In this work, we bridge the gap by addressing two key research challenges in PLL -- representation learning and label disambiguation -- in one coherent framework. Specifically, our proposed framework PiCO consists of a contrastive learning module along with a novel class prototype-based label disam…
Structure-preserving contrastive learning for spatial time series
Department of Transport\&Planning, Delft University of Technology（交通与规划系，代尔夫特理工大学）；Department of Engineering Systems and Services, Delft University of Technology（工程系统与服务系，代尔夫特理工大学）；CityAI lab, Delft University of Technology（CityAI实验室，代尔夫特理工大学）
Texas A&M University（德克萨斯A&M大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
Self-supervision is recently surging at its new frontier of graph learning. It facilitates graph representations beneficial to downstream tasks; but its success could hinge on domain knowledge for handcraft or the often expensive trials and errors. Even its state-of-the-art representative, graph contrastive learning (GraphCL), is not completely free of those needs as GraphCL uses a prefabricated prior reflected by the ad-hoc manual selection of graph data augmentations. Our work aims at advancing GraphCL by answering the following questions: How to represent the space of graph augmented views?…
UiT the Arctic University of Norway（特罗姆瑟大学-挪威北极圈大学）；Norwegian Computing Center（挪威计算中心）；University Hospital of North Norway（北挪威大学医院）
The lack of labeled data is a key challenge for learning useful representation from time series data. However, an unsupervised representation framework that is capable of producing high quality representations could be of great value. It is key to enabling transfer learning, which is especially beneficial for medical applications, where there is an abundance of data but labeling is costly and time consuming. We propose an unsupervised contrastive learning framework that is motivated from the perspective of label smoothing. The proposed approach uses a novel contrastive loss that naturally expl…
Sapienza University of Rome（罗马第一大学）；Technical University of Munich（慕尼黑工业大学）
Self-paced learning has been beneficial for tasks where some initial knowledge is available, such as weakly supervised learning and domain adaptation, to select and order the training sample sequence, from easy to complex. However its applicability remains unexplored in unsupervised learning, whereby the knowledge of the task matures during training. We propose a novel HYperbolic Self-Paced model (HYSP) for learning skeleton-based action representations. HYSP adopts self-supervision: it uses data augmentations to generate two views of the same sample, and it learns by matching one (named onlin…
Bar-Ilan University（巴伊兰大学）；Facebook AI Research（脸书人工智能研究院）
We propose a self-supervised representation learning model for the task of unsupervised phoneme boundary detection. The model is a convolutional neural network that operates directly on the raw waveform. It is optimized to identify spectral changes in the signal using the Noise-Contrastive Estimation principle. At test time, a peak detection algorithm is applied over the model outputs to produce the final boundaries. As such, the proposed model is trained in a fully unsupervised manner with no manual annotations in the form of target boundaries nor phonetic transcriptions. We compare the propo…
In this work, we introduce panoramic panoptic segmentation as the most holistic scene understanding both in terms of field of view and image level understanding for standard camera based input. A complete surrounding understanding provides a maximum of information to the agent, which is essential for any intelligent vehicle in order to make informed decisions in a safety-critical dynamic environment such as real-world traffic. In order to overcome the lack of annotated panoramic images, we propose a framework which allows model training on standard pinhole images and transfers the learned feat…
Xidian University（西安电子科技大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Embedded topic models are able to learn interpretable topics even with large and heavy-tailed vocabularies. However, they generally hold the Euclidean embedding space assumption, leading to a basic limitation in capturing hierarchical relations. To this end, we present a novel framework that introduces hyperbolic embeddings to represent words and topics. With the tree-likeness property of hyperbolic space, the underlying semantic hierarchy among words and topics can be better exploited to mine more interpretable topics. Furthermore, due to the superiority of hyperbolic geometry in representing…
Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approach
Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；ByteDance（字节跳动）；National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）
University of Science and Technology of China（中国科学技术大学）；meituan（美团）；Sichuan University（四川大学）
Learning hyperbolic embeddings for knowledge graph (KG) has gained increasing attention due to its superiority in capturing hierarchies. However, some important operations in hyperbolic space still lack good definitions, making existing methods unable to fully leverage the merits of hyperbolic space. Specifically, they suffer from two main limitations: 1) existing Graph Convolutional Network (GCN) methods in hyperbolic space rely on tangent space approximation, which would incur approximation error in representation learning, and 2) due to the lack of inner product operation definition in hype…
University of Technology Sydney（悉尼科技大学）；Monash University（莫纳什大学）；Wuhan University of Technology（武汉理工大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Unsupervised graph representation learning has emerged as a powerful tool to address real-world problems and achieves huge success in the graph learning domain. Graph contrastive learning is one of the unsupervised graph representation learning methods, which recently attracts attention from researchers and has achieved state-of-the-art performances on various tasks. The key to the success of graph contrastive learning is to construct proper contrasting pairs to acquire the underlying structural semantics of the graph. However, this key part is not fully explored currently, most of the ways ge…
UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learning
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Genomics, Chinese Academy of Sciences（中国科学院北京基因组研究所）；Peking University（北京大学）
Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection
University of Macau（澳门大学）；Zhejiang University（浙江大学）；Tsinghua University（清华大学）；Tongji University（同济大学）；University of Southampton（南安普顿大学）；University of Bologna（博洛尼亚大学）；Minzu University of China（中央民族大学）
In the realms of computer vision, it is evident that deep neural networks perform better in a supervised setting with a large amount of labeled data. The representations learned with supervision are not only of high quality but also helps the model in enhancing its accuracy. However, the collection and annotation of a large dataset are costly and time-consuming. To avoid the same, there has been a lot of research going on in the field of unsupervised visual representation learning especially in a self-supervised setting. Amongst the recent advancements in self-supervised methods for visual rec…
