School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Key Laboratory of Knowledge Engineering with Big Data (the Ministry of Education of China), Hefei University of Technology（大数据知识工程重点实验室（教育部））
Yu Wang, Lei Sang, Yi Zhang, Yiwen Zhang, Xindong Wu
Heterogeneous Graphs (HGs) effectively model complex relationships in the real world through multi-type nodes and edges. In recent years, inspired by self-supervised learning (SSL), contrastive learning (CL)-based Heterogeneous Graphs Neural Networks (HGNNs) have shown great potential in utilizing data augmentation and contrastive discriminators for downstream tasks. However, data augmentation remains limited due to the graph data's integrity. Furthermore, the contrastive discriminators suffer from sampling bias and lack local heterogeneous information. To tackle the above limitations, we propose a novel Generative-Contrastive Heterogeneous Graph Neural Network (GC-HGNN). Specifically, we propose a heterogeneous graph generative learning method that enhances CL-based paradigm. This paradigm includes: 1) A contrastive view augmentation strategy using a masked autoencoder. 2) Position-aware and semantics-aware positive sample sampling strategy for generating hard negative samples. 3) A hierarchical contrastive learning strategy aimed at capturing local and global information. Furthermore, the hierarchical contrastive learning and sampling strategies aim to constitute an enhanced contrastive discriminator under the generative-contrastive perspective. Finally, we compare our model with seventeen baselines on eight real-world datasets. Our model outperforms the latest baselines on node classification and link prediction tasks.
Heterogeneous Graph Neural Networks using Self-supervised Reciprocally Contrastive Learning
Tianjin University（天津大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Alberta（阿尔伯塔大学）
Heterogeneous graph neural networks (HGNNs) have been widely applied in heterogeneous information network tasks, while most HGNNs suffer from poor scalability or weak representation when they are applied to large-scale heterogeneous graphs. To address these problems, we propose a novel Relation-aware Heterogeneous Graph Neural Network with Contrastive Learning (RHCO) for large-scale heterogeneous graph representation learning. Unlike traditional heterogeneous graph neural networks, we adopt the contrastive learning mechanism to deal with the complex heterogeneity of large-scale heterogeneous g…
Beijing University of Posts and Telecommunications（北京邮电大学）
Heterogeneous graph neural networks (HGNNs) as an emerging technique have shown superior capacity of dealing with heterogeneous information network (HIN). However, most HGNNs follow a semi-supervised learning manner, which notably limits their wide use in reality since labels are usually scarce in real applications. Recently, contrastive learning, a self-supervised method, becomes one of the most exciting learning paradigms and shows great potential when there are no labels. In this paper, we study the problem of self-supervised HGNNs and propose a novel co-contrastive learning mechanism for H…
A wide range of models have been proposed for Graph Generative Models, necessitating effective methods to evaluate their quality. So far, most techniques use either traditional metrics based on subgraph counting, or the representations of randomly initialized Graph Neural Networks (GNNs). We propose using representations from contrastively trained GNNs, rather than random GNNs, and show this gives more reliable evaluation metrics. Neither traditional approaches nor GNN-based approaches dominate the other, however: we give examples of graphs that each approach is unable to distinguish. We demon…
CHAT: Beyond Contrastive Graph Transformer for Link Prediction in Heterogeneous Networks
Chinese Academy of Medical Sciences & Peking Union Medical College（中国医学科学院北京协和医学院）；Baidu Research（百度研究院）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Motif-aware Riemannian Graph Neural Network with Generative-Contrastive Learning
North China Electric Power University（华北电力大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beihang University（北京航空航天大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Defence Industry Secrecy Examination and Certification Center（国防工业保密审查认证中心）
University of Leeds（利兹大学）；Shaoxing University（绍兴文理学院）；Nanjing University of Information Science and Technology（南京信息工程大学）；University of Eastern Finland（东芬兰大学）；Huzhou University（湖州师范学院）
Inspired by the success of contrastive learning (CL) in computer vision and natural language processing, graph contrastive learning (GCL) has been developed to learn discriminative node representations on graph datasets. However, the development of GCL on Heterogeneous Information Networks (HINs) is still in the infant stage. For example, it is unclear how to augment the HINs without substantially altering the underlying semantics, and how to design the contrastive objective to fully capture the rich semantics. Moreover, early investigations demonstrate that CL suffers from sampling bias, wher…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Baidu Inc.（百度公司）；Griffith University（格里菲斯大学）；Xiaomi AI Lab（小米人工智能实验室）
Self-supervised learning (especially contrastive learning) methods on heterogeneous graphs can effectively get rid of the dependence on supervisory data. Meanwhile, most existing representation learning methods embed the heterogeneous graphs into a single geometric space, either Euclidean or hyperbolic. This kind of single geometric view is usually not enough to observe the complete picture of heterogeneous graphs due to their rich semantics and complex structures. Under these observations, this paper proposes a novel self-supervised learning method, termed as Geometry Contrastive Learning (GC…
Heterogeneous Graph Contrastive Learning with Spectral Augmentation
School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Unsupervised representation learning has recently received lots of interest due to its powerful generalizability through effectively leveraging large-scale unlabeled data. There are two prevalent approaches for this, contrastive learning and generative pre-training, where the former learns representations from instance-wise discrimination tasks and the latter learns them from estimating the likelihood. These seemingly orthogonal approaches have their own strengths and weaknesses. Contrastive learning tends to extract semantic information and discards details irrelevant for classifying objects,…
University of Science and Technology of China（中国科学技术大学）；Huawei Inc.（华为公司）；Shanghai Jiao Tong University（上海交通大学）；Chinese Academy of Sciences（中国科学院）
Contrastive learning has achieved great success in self-supervised visual representation learning, but existing approaches mostly ignored spatial information which is often crucial for visual representation. This paper presents heterogeneous contrastive learning (HCL), an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations. We demonstrate the effectiveness of HCL by showing that (i) it achieves higher accuracy in instance discrimination and (ii) it surpasses exis…
Zhejiang University（浙江大学）；AZFT Joint Lab for Knowledge Engine（AZFT知识引擎联合实验室）；Alibaba Group（阿里巴巴集团）
Triple extraction is an essential task in information extraction for natural language processing and knowledge graph construction. In this paper, we revisit the end-to-end triple extraction task for sequence generation. Since generative triple extraction may struggle to capture long-term dependencies and generate unfaithful triples, we introduce a novel model, contrastive triple extraction with a generative transformer. Specifically, we introduce a single shared transformer module for encoder-decoder-based generation. To generate faithful results, we propose a novel triplet contrastive trainin…
A Heterogeneous Network-based Contrastive Learning Approach for Predicting Drug-Target Interaction
College of Informatics, Huazhong Agricultural University（华中农业大学信息学院）；School of Computing, Mathematics & Engineering, Charles Sturt University（查尔斯斯特大学计算、数学与工程学院）；Artificial Intelligence & Cyber Futures Institute, Charles Sturt University（查尔斯斯特大学人工智能与网络未来研究院）；Faculty of Engineering, University of Mines and Technology（矿业技术大学工程学院）；School of Mathematics and Computer Science, Xinyu University（新余学院数学与计算机学院）
Contrastive learning and supervised learning have both seen significant progress and success. However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network. In this paper we show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning. Beyond presenting this unified view, we show our specific choice of approximation of the energy-based loss outperforms the existing practice in terms of classification…
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computing, Macquarie University（麦考瑞大学计算学院）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）；Faculty of Engineering and Information Technology, University of Technology Sydney（悉尼科技大学工程与信息技术学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University at Buffalo（布法罗大学）；MIT-IBM Watson AI Lab, IBM Research（麻省理工学院-IBM沃森人工智能实验室，IBM研究院）
With the advent of big data across multiple high-impact applications, we are often facing the challenge of complex heterogeneity. The newly collected data usually consist of multiple modalities and are characterized with multiple labels, thus exhibiting the co-existence of multiple types of heterogeneity. Although state-of-the-art techniques are good at modeling complex heterogeneity with sufficient label information, such label information can be quite expensive to obtain in real applications. Recently, researchers pay great attention to contrastive learning due to its prominent performance b…
Towards Performance-Enhanced Model-Contrastive Federated Learning using Historical Information in Heterogeneous Scenarios
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)（计算能力网络与信息安全重点实验室，教育部，山东计算机科学中心，齐鲁工业大学（山东科学院））；School of Information and Software Engineering, University of Electronic Science and Technology of China（信息与软件工程学院，电子科技大学）；School of Computer Science and Engineering, Shandong University of Science and Technology（计算机科学与工程学院，山东科技大学）；School of Big Data and Software Engineering, Chongqing University（大数据与软件工程学院，重庆大学）
La Trobe University（乐卓博大学）；Monash University（蒙纳士大学）
Anomaly detection from graph data has drawn much attention due to its practical significance in many critical applications including cybersecurity, finance, and social networks. Existing data mining and machine learning methods are either shallow methods that could not effectively capture the complex interdependency of graph data or graph autoencoder methods that could not fully exploit the contextual information as supervision signals for effective anomaly detection. To overcome these challenges, in this paper, we propose a novel method, Self-Supervised Learning for Graph Anomaly Detection (S…
Contrastive Balancing Representation Learning for Heterogeneous Dose-Response Curves Estimation
Zhejiang University（浙江大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Peking University（北京大学）；Emory University（埃默里大学）；Tsinghua University（清华大学）；Didi Chuxing（滴滴出行）
Tsinghua University（清华大学）；Tsinghua National Laboratory for Information Science and Technology (TNList)（清华信息科学与技术国家实验室）；Anhui University（安徽大学）；Renmin University of China（中国人民大学）
Deep supervised learning has achieved great success in the last decade. However, its deficiencies of dependence on manual labels and vulnerability to attacks have driven people to explore a better solution. As an alternative, self-supervised learning attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vis…
Adversarial Training (AT) is known as an effective approach to enhance the robustness of deep neural networks. Recently researchers notice that robust models with AT have good generative ability and can synthesize realistic images, while the reason behind it is yet under-explored. In this paper, we demystify this phenomenon by developing a unified probabilistic framework, called Contrastive Energy-based Models (CEM). On the one hand, we provide the first probabilistic characterization of AT through a unified understanding of robustness and generative ability. On the other hand, our unified fra…
A Generative Graph Contrastive Learning Model with Global Signal
College of Han Hong, Southwest University, Chongqing, China（Honghan College, Southwest University）；College of Computer and Information Science, Southwest University, Chongqing, China（Computer and Information Science College, Southwest University）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Texas A&M University（德州农工大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper targets at improving the generalizability of hypergraph neural networks in the low-label regime, through applying the contrastive learning approach from images/graphs (we refer to it as HyperGCL). We focus on the following question: How to construct contrastive views for hypergraphs via augmentations? We provide the solutions in two folds. First, guided by domain knowledge, we fabricate two schemes to augment hyperedges with higher-order relations encoded, and adopt three vertex augmentation strategies from graph-structured data. Second, in search of more effective views in a data-d…
In this paper, we present a framework for contrastive learning for audio representations, in a self supervised frame work without access to any ground truth labels. The core idea in self supervised contrastive learning is to map an audio signal and its various augmented versions (representative of salient aspects of audio like pitch, timbre etc.) to a space where they are close together, and are separated from other different signals. In addition we also explore generative models based on state of the art transformer based architectures for learning latent spaces for audio signals, without acc…
Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision
Zhejiang University of Technology（浙江工业大学）；Binjiang Institute of Artificial Intelligence, ZJUT（浙江工业大学滨江人工智能研究院）；Science and Technology on Communication Information Security Control Laboratory（通信信息安全控制技术实验室）
LAMP: Learnable Meta-Path Guided Adversarial Contrastive Learning for Heterogeneous Graphs
University of New South Wales（新南威尔士大学）；Yonsei University（延世大学）；RIKEN AIP（理化学研究所人工智能项目）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
CoSD: Collaborative Stance Detection with Contrastive Heterogeneous Topic Graph Learning
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Department of Computer Science, Tongji University（同济大学计算机科学系）；College of Computer Science and Technology, Taiyuan University of Technology（太原理工大学计算机科学与技术学院）
CoopHash: Cooperative Learning of Multipurpose Descriptor and Contrastive Pair Generator via Variational MCMC Teaching for Supervised Image Hashing
College of Engineering and Computer Science, VinUniversity（VinUniversity工程与计算机科学学院）；Akool Research（Akool研究院）；Department of Statistics and Data Science, UCLA（加州大学洛杉矶分校统计与数据科学系）；Google Research（谷歌研究院）；VecML
针对监督图像哈希标注不足及 GAN 难联合训练的问题，提出基于能量协同学习的 CoopHash，联合训练对比图像生成器与多用途描述器，取得最高 10% 相对提升。
University of Tübingen（蒂宾根大学）；IMPRS for Intelligent Systems（智能系统马克斯·普朗克国际研究学院）；EPFL（洛桑联邦理工学院）
Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory…
