Contrastive learning has achieved great success in self-supervised visual representation learning, but existing approaches mostly ignored spatial information which is often crucial for visual representation. This paper presents heterogeneous contrastive learning (HCL), an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations. We demonstrate the effectiveness of HCL by showing that (i) it achieves higher accuracy in instance discrimination and (ii) it surpasses existing pre-training methods in a series of downstream tasks while shrinking the pre-training costs by half. More importantly, we show that our approach achieves higher efficiency in visual representations, and thus delivers a key message to inspire the future research of self-supervised visual representation learning.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University at Buffalo（布法罗大学）；MIT-IBM Watson AI Lab, IBM Research（麻省理工学院-IBM沃森人工智能实验室，IBM研究院）
With the advent of big data across multiple high-impact applications, we are often facing the challenge of complex heterogeneity. The newly collected data usually consist of multiple modalities and are characterized with multiple labels, thus exhibiting the co-existence of multiple types of heterogeneity. Although state-of-the-art techniques are good at modeling complex heterogeneity with sufficient label information, such label information can be quite expensive to obtain in real applications. Recently, researchers pay great attention to contrastive learning due to its prominent performance b…
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Defence Industry Secrecy Examination and Certification Center（国防工业保密审查认证中心）
University of Leeds（利兹大学）；Shaoxing University（绍兴文理学院）；Nanjing University of Information Science and Technology（南京信息工程大学）；University of Eastern Finland（东芬兰大学）；Huzhou University（湖州师范学院）
Inspired by the success of contrastive learning (CL) in computer vision and natural language processing, graph contrastive learning (GCL) has been developed to learn discriminative node representations on graph datasets. However, the development of GCL on Heterogeneous Information Networks (HINs) is still in the infant stage. For example, it is unclear how to augment the HINs without substantially altering the underlying semantics, and how to design the contrastive objective to fully capture the rich semantics. Moreover, early investigations demonstrate that CL suffers from sampling bias, wher…
Medical report generation is a challenging task since it is time-consuming and requires expertise from experienced radiologists. The goal of medical report generation is to accurately capture and describe the image findings. Previous works pretrain their visual encoding neural networks with large datasets in different domains, which cannot learn general visual representation in the specific medical domain. In this work, we propose a medical report generation framework that uses a contrastive learning approach to pretrain the visual encoder and requires no additional meta information. In additi…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Baidu Inc.（百度公司）；Griffith University（格里菲斯大学）；Xiaomi AI Lab（小米人工智能实验室）
Self-supervised learning (especially contrastive learning) methods on heterogeneous graphs can effectively get rid of the dependence on supervisory data. Meanwhile, most existing representation learning methods embed the heterogeneous graphs into a single geometric space, either Euclidean or hyperbolic. This kind of single geometric view is usually not enough to observe the complete picture of heterogeneous graphs due to their rich semantics and complex structures. Under these observations, this paper proposes a novel self-supervised learning method, termed as Geometry Contrastive Learning (GC…
Heterogeneous Graph Contrastive Learning with Spectral Augmentation
School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset
Department of Computer Science（计算机科学系）；Brown University（布朗大学）；Institute for Bioinformatics and Medical Informatics（生物信息学与医学信息学研究所）；University of Tübingen（图宾根大学）；Center for Computational Molecular Biology（计算分子生物学中心）
A Heterogeneous Network-based Contrastive Learning Approach for Predicting Drug-Target Interaction
College of Informatics, Huazhong Agricultural University（华中农业大学信息学院）；School of Computing, Mathematics & Engineering, Charles Sturt University（查尔斯斯特大学计算、数学与工程学院）；Artificial Intelligence & Cyber Futures Institute, Charles Sturt University（查尔斯斯特大学人工智能与网络未来研究院）；Faculty of Engineering, University of Mines and Technology（矿业技术大学工程学院）；School of Mathematics and Computer Science, Xinyu University（新余学院数学与计算机学院）
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computing, Macquarie University（麦考瑞大学计算学院）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）；Faculty of Engineering and Information Technology, University of Technology Sydney（悉尼科技大学工程与信息技术学院）
College of Mathematics and Informatics, South China Agricultural University（华南农业大学数学与信息学院）；Key Laboratory of Smart Agricultural Technology in Tropical South China, Ministry of Agriculture and Rural Affairs（农业农村部华南热带智慧农业技术重点实验室）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机学院）
Contrastive deep clustering has recently gained significant attention with its ability of joint contrastive learning and clustering via deep neural networks. Despite the rapid progress, previous works mostly require both positive and negative sample pairs for contrastive clustering, which rely on a relative large batch-size. Moreover, they typically adopt a two-stream architecture with two augmented views, which overlook the possibility and potential benefits of multi-stream architectures (especially with heterogeneous or hybrid networks). In light of this, this paper presents a new end-to-end…
Contrastive Balancing Representation Learning for Heterogeneous Dose-Response Curves Estimation
Zhejiang University（浙江大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Peking University（北京大学）；Emory University（埃默里大学）；Tsinghua University（清华大学）；Didi Chuxing（滴滴出行）
PepHarmony: A Multi-View Contrastive Learning Framework for Integrated Sequence and Structure-Based Peptide Encoding
Jilin University（吉林大学）；Syneron Technology；Beijing Life Science Academy（北京生命科学学院）；the University of Hong Kong（香港大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
LAMP: Learnable Meta-Path Guided Adversarial Contrastive Learning for Heterogeneous Graphs
University of New South Wales（新南威尔士大学）；Yonsei University（延世大学）；RIKEN AIP（理化学研究所人工智能项目）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University（卡内基梅隆大学）；Tencent（腾讯）
Graphs are powerful representations for relations among objects, which have attracted plenty of attention. A fundamental challenge for graph learning is how to train an effective Graph Neural Network (GNN) encoder without labels, which are expensive and time consuming to obtain. Contrastive Learning (CL) is one of the most popular paradigms to address this challenge, which trains GNNs by discriminating positive and negative node pairs. Despite the success of recent CL methods, there are still two under-explored problems. First, how to reduce the semantic error introduced by random topology bas…
ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data
Lane Dept. of Computer Science and Electrical Engineering（兰恩计算机科学与电气工程系）；Dept. of Geology & Geography（地质与地理系）；West Virginia University（西弗吉尼亚大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Green University of Bangladesh（孟加拉国绿色大学）；Stamford University Bangladesh（班加罗尔斯坦福大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Emory University（埃默里大学）
Recently, heterogeneous Graph Neural Networks (GNNs) have become a de facto model for analyzing HGs, while most of them rely on a relative large number of labeled data. In this work, we investigate Contrastive Learning (CL), a key component in self-supervised approaches, on HGs to alleviate the label scarcity problem. We first generate multiple semantic views according to metapaths and network schemas. Then, by pushing node embeddings corresponding to different semantic views close to each other (positives) and pulling other embeddings apart (negatives), one can obtain informative representati…
National University of Defense Technology（国防科技大学）；Southeast University（东南大学）；Purple Mountain Laboratories（紫金山实验室）
Similarity metric is crucial for massive MIMO positioning utilizing channel state information~(CSI). In this letter, we propose a novel massive MIMO CSI similarity learning method via deep convolutional neural network~(DCNN) and contrastive learning. A contrastive loss function is designed considering multiple positive and negative CSI samples drawn from a training dataset. The DCNN encoder is trained using the loss so that positive samples are mapped to points close to the anchor's encoding, while encodings of negative samples are kept away from the anchor's in the representation space. Evalu…
Jointly Learning Representations for Map Entities via Heterogeneous Graph Contrastive Learning
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Economics and Management, Beihang University（北京航空航天大学经济管理学院）
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Heterogeneous graph neural networks can represent information of heterogeneous graphs with excellent ability. Recently, self-supervised learning manner is researched which learns the unique expression of a graph through a contrastive learning method. In the absence of labels, this learning methods show great potential. However, contrastive learning relies heavily on positive and negative pairs, and generating high-quality pairs from heterogeneous graphs is difficult. In this paper, in line with recent innovations in self-supervised learning called BYOL or bootstrapping, we introduce a that can…
We propose methods to strengthen the invariance properties of representations obtained by contrastive learning. While existing approaches implicitly induce a degree of invariance as representations are learned, we look to more directly enforce invariance in the encoding process. To this end, we first introduce a training objective for contrastive learning that uses a novel regularizer to control how the representation changes under transformation. We show that representations trained with this objective perform better on downstream tasks and are more robust to the introduction of nuisance tran…
Federated Contrastive Learning for Personalized Semantic Communication
Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；University of Essex（埃塞克斯大学）；Peng Cheng Laboratory（鹏城实验室）；Queen Mary University of London（伦敦玛丽女王大学）
Autodesk AI Lab（欧特克人工智能实验室）；Vector Institute（向量研究所）
We introduce a self-supervised approach for learning node and graph level representations by contrasting structural views of graphs. We show that unlike visual representation learning, increasing the number of views to more than two or contrasting multi-scale encodings do not improve performance, and the best performance is achieved by contrasting encodings from first-order neighbors and a graph diffusion. We achieve new state-of-the-art results in self-supervised learning on 8 out of 8 node and graph classification benchmarks under the linear evaluation protocol. For example, on Cora (node) a…
FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning
Shanghai Jiao Tong University（上海交通大学）；Institute for AI Industry Research, Tsinghua University（清华大学智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Queen’s University Belfast（英国女王大学）
AS-GCL: Asymmetric Spectral Augmentation on Graph Contrastive Learning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyberspace Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Renmin University of China（中国人民大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Graph representation learning has attracted a surge of interest recently, whose target at learning discriminant embedding for each node in the graph. Most of these representation methods focus on supervised learning and heavily depend on label information. However, annotating graphs are expensive to obtain in the real world, especially in specialized domains (i.e. biology), as it needs the annotator to have the domain knowledge to label the graph. To approach this problem, self-supervised learning provides a feasible solution for graph representation learning. In this paper, we propose a Multi…
Genentech Research and Early Development（基因泰克研究与早期开发部）；Stanford University（斯坦福大学）
Molecular shape and geometry dictate key biophysical recognition processes, yet many graph neural networks disregard 3D information for molecular property prediction. Here, we propose a new contrastive-learning procedure for graph neural networks, Molecular Contrastive Learning from Shape Similarity (MolCLaSS), that implicitly learns a three-dimensional representation. Rather than directly encoding or targeting three-dimensional poses, MolCLaSS matches a similarity objective based on Gaussian overlays to learn a meaningful representation of molecular shape. We demonstrate how this framework na…
EAGLE: Contrastive Learning for Efficient Graph Anomaly Detection
Institute of Innovation, Science and Sustainability, Federation University Australia（创新、科学与可持续性研究所，联邦大学澳大利亚）；School of Software, Dalian University of Technology（软件学院，大连理工大学）；Computing Center, Anshan Normal University（计算机中心，鞍山师范学院）
Preferred Networks, Inc.（Preferred Networks公司）；University of Oxford（牛津大学）
In contrastive representation learning, data representation is trained so that it can classify the image instances even when the images are altered by augmentations. However, depending on the datasets, some augmentations can damage the information of the images beyond recognition, and such augmentations can result in collapsed representations. We present a partial solution to this problem by formalizing a stochastic encoding process in which there exist a tug-of-war between the data corruption introduced by the augmentations and the information preserved by the encoder. We show that, with the …
Advertising click fraud detection plays one of the vital roles in current E-commerce websites as advertising is an essential component of its business model. It aims at, given a set of corresponding features, e.g., demographic information of users and statistical features of clicks, predicting whether a click is fraudulent or not in the community. Recent efforts attempted to incorporate attributed behavior sequence and heterogeneous network for extracting complex features of users and achieved significant effects on click fraud detection. In this paper, we propose a Multimodal and Contrastive …
North University of China（中北大学）；Dalian University of Technology（大连理工大学）
Predicting medications is a crucial task in many intelligent healthcare systems. It can assist doctors in making informed medication decisions for patients according to electronic medical records (EMRs). However, medication prediction is a challenging data mining task due to the complex relations between medical codes. Most existing studies focus on utilizing inherent relations between homogeneous codes of medical ontology graph to enhance their representations using supervised methods, and few studies pay attention to the valuable relations between heterogeneous or homogeneous medical codes f…
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples' positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example's loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient…
There have been several recent efforts towards developing representations for multivariate time-series in an unsupervised learning framework. Such representations can prove beneficial in tasks such as activity recognition, health monitoring, and anomaly detection. In this paper, we consider a setting where we observe time-series at each node in a dynamic graph. We propose a framework called GraphTNC for unsupervised learning of joint representations of the graph and the time-series. Our approach employs a contrastive learning strategy. Based on an assumption that the time-series and graph evol…
North China Electric Power University（华北电力大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beihang University（北京航空航天大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Alibaba Group（阿里巴巴集团）；Renmin University of China（中国人民大学）
Multi-scenario ad ranking aims at leveraging the data from multiple domains or channels for training a unified ranking model to improve the performance at each individual scenario. Although the research on this task has made important progress, it still lacks the consideration of cross-scenario relations, thus leading to limitation in learning capability and difficulty in interrelation modeling. In this paper, we propose a Hybrid Contrastive Constrained approach (HC^2) for multi-scenario ad ranking. To enhance the modeling of data interrelation, we elaborately design a hybrid contrastive learn…
