Self-supervised Consensus Representation Learning for Attributed Graph
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；The Noah’s Ark Lab, Huawei Technologies Company Limited（华为技术有限公司诺亚方舟实验室）；School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）
Attempting to fully exploit the rich information of topological structure and node features for attributed graph, we introduce self-supervised learning mechanism to graph representation learning and propose a novel Self-supervised Consensus Representation Learning (SCRL) framework. In contrast to most existing works that only explore one graph, our proposed SCRL method treats graph from two perspectives: topology graph and feature graph. We argue that their embeddings should share some common information, which could serve as a supervisory signal. Specifically, we construct the feature graph of node features via k-nearest neighbor algorithm. Then graph convolutional network (GCN) encoders extract features from two graphs respectively. Self-supervised loss is designed to maximize the agreement of the embeddings of the same node in the topology graph and the feature graph. Extensive experiments on real citation networks and social networks demonstrate the superiority of our proposed SCRL over the state-of-the-art methods on semi-supervised node classification task. Meanwhile, compared with its main competitors, SCRL is rather efficient.
Bitdefender（Bitdefender公司）；University Politehnica of Bucharest（布加勒斯特理工大学）；University of Bucharest（布加勒斯特大学）
The human ability to synchronize the feedback from all their senses inspired recent works in multi-task and multi-modal learning. While these works rely on expensive supervision, our multi-task graph requires only pseudo-labels from expert models. Every graph node represents a task, and each edge learns between tasks transformations. Once initialized, the graph learns self-supervised, based on a novel consensus shift algorithm that intelligently exploits the agreement between graph pathways to generate new pseudo-labels for the next learning cycle. We demonstrate significant improvement from o…
Decoupling anomaly discrimination and representation learning: self-supervised learning for anomaly detection on attributed graph
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Software Engineering, Sun Yat-sen University（中山大学软件工程学院）；School of System Science and Engineering, Sun Yat-sen University（中山大学系统科学与工程学院）；Merchants Union Consumer Finance Co., Ltd（招商局联融消费金融股份有限公司）
Arizona State University（亚利桑那州立大学）；Texas A&M University（得克萨斯农工大学）；Michigan State University（密歇根州立大学）；Huawei Research America（华为美国研究院）
Network embedding leverages the node proximity manifested to learn a low-dimensional node vector representation for each node in the network. The learned embeddings could advance various learning tasks such as node classification, network clustering, and link prediction. Most, if not all, of the existing works, are overwhelmingly performed in the context of plain and static networks. Nonetheless, in reality, network structure often evolves over time with addition/deletion of links and nodes. Also, a vast majority of real-world networks are associated with a rich set of node attributes, and the…
Consensus maximisation learning can provide self-supervision when different views are available of the same data. The distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present a new self-supervised learning framework for learning sentence representations which minimises the disagreement between two views of the same sentence where one view encodes the sentence with a recur…
Hong Kong Baptist University（香港浸会大学）；CeMM Research Center for Molecular Medicine of the Austrian Academy of Sciences（奥地利科学院分子医学CeMM研究中心）；Universität Vienna（维也纳大学）；Shenzhen University（深圳大学）
Monash University（莫纳什大学）；Alibaba Group（阿里巴巴集团）；Nanjing University of Science and Technology（南京理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）；University of Minnesota（明尼苏达大学）
Anomaly detection on attributed networks attracts considerable research interests due to wide applications of attributed networks in modeling a wide range of complex systems. Recently, the deep learning-based anomaly detection methods have shown promising results over shallow approaches, especially on networks with high-dimensional attributes and complex structures. However, existing approaches, which employ graph autoencoder as their backbone, do not fully exploit the rich information of the network, resulting in suboptimal performance. Furthermore, these methods do not directly target anomal…
University of Electronic Science and Technology of China（电子科技大学）
With the explosive growth of information technology, multi-view graph data have become increasingly prevalent and valuable. Most existing multi-view clustering techniques either focus on the scenario of multiple graphs or multi-view attributes. In this paper, we propose a generic framework to cluster multi-view attributed graph data. Specifically, inspired by the success of contrastive learning, we propose multi-view contrastive graph clustering (MCGC) method to learn a consensus graph since the original graph could be noisy or incomplete and is not directly applicable. Our method composes of …
Recent advances in deep clustering and unsupervised representation learning are based on the idea that different views of an input image (generated through data augmentation techniques) must either be closer in the representation space, or have a similar cluster assignment. Bootstrap Your Own Latent (BYOL) is one such representation learning algorithm that has achieved state-of-the-art results in self-supervised image classification on ImageNet under the linear evaluation protocol. However, the utility of the learnt features of BYOL to perform clustering is not explored. In this work, we study…
IPM-FM: A Foundation Model with Consensus Feature Selection for Industrial Process Monitoring
University of British Columbia（不列颠哥伦比亚大学）；Jiangxi University of Finance and Economics（江西财经大学）；Chongqing University（重庆大学）；Institute for Infocomm Research（信息通信研究院）；Nanyang Technological University（南洋理工大学）
HyperFuse: Fast Self-Supervised Node Embeddings for Attributed Hypergraphs
Indian Institute of Science Education and Research Thiruvananthapuram（印度科学教育与研究学院蒂鲁文南特普拉姆分校）；Indian Institute of Science Education and Research Kolkata（印度科学教育与研究学院加尔各答分校）；University of Rostock（罗斯托克大学）；Stellenbosch Institute for Advanced Study (STIAS)（斯泰伦博斯高等研究院）
Multi-view attributed graph clustering is an important approach to partition multi-view data based on the attribute feature and adjacent matrices from different views. Some attempts have been made in utilizing Graph Neural Network (GNN), which have achieved promising clustering performance. Despite this, few of them pay attention to the inherent specific information embedded in multiple views. Meanwhile, they are incapable of recovering the latent high-level representation from the low-level ones, greatly limiting the downstream clustering performance. To fill these gaps, a novel Dual Informat…
Beyond the Academic Monoculture: A Unified Framework and Industrial Perspective for Attributed Graph Clustering
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Ant Group（蚂蚁集团）；Westlake University（西湖大学）；College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）
It is a consensus that small models perform quite poorly under the paradigm of self-supervised contrastive learning. Existing methods usually adopt a large off-the-shelf model to transfer knowledge to the small one via distillation. Despite their effectiveness, distillation-based methods may not be suitable for some resource-restricted scenarios due to the huge computational expenses of deploying a large model. In this paper, we study the issue of training self-supervised small models without distillation signals. We first evaluate the representation spaces of the small models and make two non…
Data Science Institute, University of Technology Sydney（悉尼科技大学数据科学研究所）；School of Information Technology, Deakin University（迪肯大学信息技术学院）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）；School of Computer Science, University of Technology Sydney（悉尼科技大学计算机科学学院）
Recent works for attributed network clustering utilize graph convolution to obtain node embeddings and simultaneously perform clustering assignments on the embedding space. It is effective since graph convolution combines the structural and attributive information for node embedding learning. However, a major limitation of such works is that the graph convolution only incorporates the attribute information from the local neighborhood of nodes but fails to exploit the mutual affinities between nodes and attributes. In this regard, we propose a variational co-embedding learning model for attribu…
Monash University（莫纳什大学）；The University of Sydney（悉尼大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Link prediction aims to infer the link existence between pairs of nodes in networks/graphs. Despite their wide application, the success of traditional link prediction algorithms is hindered by three major challenges -- link sparsity, node attribute noise and dynamic changes -- that are faced by many real-world networks. To address these challenges, we propose a Contextualized Self-Supervised Learning (CSSL) framework that fully exploits structural context prediction for link prediction. The proposed CSSL framework learns a link encoder to infer the link existence probability from paired node e…
Self-supervised pretraining is the method of choice for natural language processing models and is rapidly gaining popularity in many vision tasks. Recently, self-supervised pretraining has shown to outperform supervised pretraining for many downstream vision applications, marking a milestone in the area. This superiority is attributed to the negative impact of incomplete labelling of the training images, which convey multiple concepts, but are annotated using a single dominant class label. Although Self-Supervised Learning (SSL), in principle, is free of this limitation, the choice of pretext …
The success of deep learning in medical imaging is mostly achieved at the cost of a large labeled data set. Semi-supervised learning (SSL) provides a promising solution by leveraging the structure of unlabeled data to improve learning from a small set of labeled data. Self-ensembling is a simple approach used in SSL to encourage consensus among ensemble predictions of unknown labels, improving generalization of the model by making it more insensitive to the latent space. Currently, such an ensemble is obtained by randomization such as dropout regularization and random data augmentation. In thi…
Self-supervised contrastive learning performs non-linear system identification
Institute of Computational Biology, Computational Health Center, Helmholtz Munich and Munich Center for Machine Learning (MCML)（计算生物学研究所、计算健康中心、海德堡慕尼黑和慕尼黑机器学习中心）
Sun Yat-sen University（中山大学）；University of Glasgow（格拉斯哥大学）
Deep generative models (DGMs) have achieved remarkable advances. Semi-supervised variational auto-encoders (SVAE) as a classical DGM offer a principled framework to effectively generalize from small labelled data to large unlabelled ones, but it is difficult to incorporate rich unstructured relationships within the multiple heterogeneous entities. In this paper, to deal with the problem, we present a semi-supervised co-embedding model for attributed networks (SCAN) based on the generalized SVAE for heterogeneous data, which collaboratively learns low-dimensional vector representations of both …
Viterbi School of Engineering（维特比工程学院）；University of Southern California（南加利福尼亚大学）
Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data privacy. An essential but rarely studied challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL compared to centralized training due to the fact that FL users are often reluctant to label their private data. Furthermore, due to the heterogeneous nature of the data at edge devices, it is crucial to develop personalized models. In this paper we propose self-supervised federated learni…
In this paper, we propose a novel framework, called Semi-supervised Embedding in Attributed Networks with Outliers (SEANO), to learn a low-dimensional vector representation that systematically captures the topological proximity, attribute affinity and label similarity of vertices in a partially labeled attributed network (PLAN). Our method is designed to work in both transductive and inductive settings while explicitly alleviating noise effects from outliers. Experimental results on various datasets drawn from the web, text and image domains demonstrate the advantages of SEANO over state-of-th…
The past few years have witnessed a remarkable advance in deep learning for EEG-based sleep stage classification (SSC). However, the success of these models is attributed to possessing a massive amount of labeled data for training, limiting their applicability in real-world scenarios. In such scenarios, sleep labs can generate a massive amount of data, but labeling these data can be expensive and time-consuming. Recently, the self-supervised learning (SSL) paradigm has shined as one of the most successful techniques to overcome the scarcity of labeled data. In this paper, we evaluate the effic…
The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所先进智能项目中心）
Contrastive representation learning encourages data representation to make semantically similar pairs closer than randomly drawn negative samples, which has been successful in various domains such as vision, language, and graphs. Recent theoretical studies have attempted to explain the benefit of the large negative sample size by upper-bounding the downstream classification loss with the contrastive loss. However, the previous surrogate bounds have two drawbacks: they are only legitimate for a limited range of negative sample sizes and prohibitively large even within that range. Due to these d…
UniGLM: Training One Unified Language Model for Text-Attributed Graph Embedding
New York University Shanghai（上海纽约大学）；Bakar Computational Health Sciences Institute（贝克计算健康科学研究所）；University of California San Francisco（加利福尼亚大学旧金山分校）；University of Georgia（佐治亚大学）
Text-Attributed Graph Anomaly Detection via Multi-Scale Cross- and Uni-Modal Contrastive Learning
School of Computer Science and Technology, Xi'an Jiaotong University（西安交通大学计算机科学与技术学院）；Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi’an Jiaotong University（陕西省大数据知识工程重点实验室）；School of Distance Education, Xi’an Jiaotong University（西安交通大学继续教育学院）；University of Virginia（弗吉尼亚大学）
Hybrid-Collaborative Augmentation and Contrastive Sample Adaptive-Differential Awareness for Robust Attributed Graph Clustering
College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）；Beijing University of Technology（北京工业大学）；The University of Sydney Business School, The University of Sydney（悉尼大学商学院）
Recent years have witnessed an upsurge of interest in the problem of anomaly detection on attributed networks due to its importance in both research and practice. Although various approaches have been proposed to solve this problem, two major limitations exist: (1) unsupervised approaches usually work much less efficiently due to the lack of supervisory signal, and (2) existing anomaly detection methods only use local contextual information to detect anomalous nodes, e.g., one- or two-hop information, but ignore the global contextual information. Since anomalous nodes differ from normal nodes …
University of Electronic Science and Technology of China（电子科技大学）
Graph-based multi-view clustering has achieved better performance than most non-graph approaches. However, in many real-world scenarios, the graph structure of data is not given or the quality of initial graph is poor. Additionally, existing methods largely neglect the high-order neighborhood information that characterizes complex intrinsic interactions. To tackle these problems, we introduce an approach called high-order multi-view clustering (HMvC) to explore the topology structure information of generic data. Firstly, graph filtering is applied to encode structure information, which unifies…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We present InfoMotif, a new semi-supervised, motif-regularized, learning framework over graphs. We overcome two key limitations of message passing in popular graph neural networks (GNNs): localization (a k-layer GNN cannot utilize features outside the k-hop neighborhood of the labeled training nodes) and over-smoothed (structurally indistinguishable) representations. We propose the concept of attributed structural roles of nodes based on their occurrence in different network motifs, independent of network proximity. Two nodes share attributed structural roles if they participate in topological…
Anomaly detection on the attributed network has recently received increasing attention in many research fields, such as cybernetic anomaly detection and financial fraud detection. With the wide application of deep learning on graph representations, existing approaches choose to apply euclidean graph encoders as their backbone, which may lose important hierarchical information, especially in complex networks. To tackle this problem, we propose an efficient anomaly detection framework using hyperbolic self-supervised contrastive learning. Specifically, we first conduct the data augmentation by p…
Arizona State University（亚利桑那州立大学）；Texas A&M University（得克萨斯农工大学）；University of Virginia（弗吉尼亚大学）；Illinois Institute of Technology（伊利诺伊理工学院）；Singapore Management University（新加坡管理大学）
Attributed networks nowadays are ubiquitous in a myriad of high-impact applications, such as social network analysis, financial fraud detection, and drug discovery. As a central analytical task on attributed networks, node classification has received much attention in the research community. In real-world attributed networks, a large portion of node classes only contain limited labeled instances, rendering a long-tail node class distribution. Existing node classification algorithms are unequipped to handle the \textit{few-shot} node classes. As a remedy, few-shot learning has attracted a surge…
Indian Institute of Technology Madras（马德拉斯印度理工学院）；University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；Ohio State University（俄亥俄州立大学）
Semi-supervised node classification in attributed graphs, i.e., graphs with node features, involves learning to classify unlabeled nodes given a partially labeled graph. Label predictions are made by jointly modeling the node and its' neighborhood features. State-of-the-art models for node classification on such attributed graphs use differentiable recursive functions that enable aggregation and filtering of neighborhood information from multiple hops. In this work, we analyze the representation capacity of these models to regulate information from multiple hops independently. From our analysi…
Nanjing University of Posts and Telecommunications（南京邮电大学）
The pre-training on the graph neural network model can learn the general features of large-scale networks or networks of the same type by self-supervised methods, which allows the model to work even when node labels are missing. However, the existing pre-training methods do not take network evolution into consideration. This paper proposes a pre-training method on dynamic graph neural networks (PT-DGNN), which uses dynamic attributed graph generation tasks to simultaneously learn the structure, semantics, and evolution features of the graph. The method includes two steps: 1) dynamic sub-graph …
