Unsupervised graph representation learning is critical to a wide range of applications where labels may be scarce or expensive to procure. Contrastive learning (CL) is an increasingly popular paradigm for such settings and the state-of-the-art in unsupervised visual representation learning. Recent work attributes the success of visual CL to use of task-relevant augmentations and large, diverse datasets. Interestingly, graph CL frameworks report strong performance despite using orders of magnitude smaller datasets and employing domain-agnostic graph augmentations (DAGAs). Motivated by this discrepancy, we probe the quality of representations learnt by popular graph CL frameworks using DAGAs. We find that DAGAs can destroy task-relevant information and harm the model's ability to learn discriminative representations. On small benchmark datasets, we show the inductive bias of graph neural networks can significantly compensate for this weak discriminability. Based on our findings, we propose several sanity checks that enable practitioners to quickly assess the quality of their model's learned representations. We further propose a broad strategy for designing task-aware augmentations that are amenable to graph CL and demonstrate its efficacy on two large-scale, complex graph applications. For example, in graph-based document classification, we show task-aware augmentations improve accuracy up to 20%.
Deep learning on graphs has recently achieved remarkable success on a variety of tasks, while such success relies heavily on the massive and carefully labeled data. However, precise annotations are generally very expensive and time-consuming. To address this problem, self-supervised learning (SSL) is emerging as a new paradigm for extracting informative knowledge through well-designed pretext tasks without relying on manual labels. In this survey, we extend the concept of SSL, which first emerged in the fields of computer vision and natural language processing, to present a timely and comprehe…
Tsinghua University（清华大学）；Institute for AI Industry Research, Tsinghua University（清华大学智能产业研究院）；Harbin Institute of Technology（哈尔滨工业大学）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）
Contrastive learning (CL) has recently been applied to adversarial learning tasks. Such practice considers adversarial samples as additional positive views of an instance, and by maximizing their agreements with each other, yields better adversarial robustness. However, this mechanism can be potentially flawed, since adversarial perturbations may cause instance-level identity confusion, which can impede CL performance by pulling together different instances with separate identities. To address this issue, we propose to treat adversarial samples unequally when contrasted, with an asymmetric Inf…
Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding
Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Hong Kong University of Science and Technology（香港科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Ghent University – imec（根特大学–imec）；Ghent University Hospital（根特大学医院）；Analytical Minds Ltd.（Analytical Minds有限公司）
Information extracted from electrohysterography recordings could potentially prove to be an interesting additional source of information to estimate the risk on preterm birth. Recently, a large number of studies have reported near-perfect results to distinguish between recordings of patients that will deliver term or preterm using a public resource, called the Term/Preterm Electrohysterogram database. However, we argue that these results are overly optimistic due to a methodological flaw being made. In this work, we focus on one specific type of methodological flaw: applying over-sampling befo…
Deep models trained in supervised mode have achieved remarkable success on a variety of tasks. When labeled samples are limited, self-supervised learning (SSL) is emerging as a new paradigm for making use of large amounts of unlabeled samples. SSL has achieved promising performance on natural language and image learning tasks. Recently, there is a trend to extend such success to graph data using graph neural networks (GNNs). In this survey, we provide a unified review of different ways of training GNNs using SSL. Specifically, we categorize SSL methods into contrastive and predictive models. I…
Annotation noise is widespread in datasets, but manually revising a flawed corpus is time-consuming and error-prone. Hence, given the prior knowledge in Pre-trained Language Models and the expected uniformity across all annotations, we attempt to reduce annotation noise in the corpus through two tasks automatically: (1) Annotation Inconsistency Detection that indicates the credibility of annotations, and (2) Annotation Error Correction that rectifies the abnormal annotations. We investigate how to acquire semantic sensitive annotation representations from Pre-trained Language Models, expecting…
Overcoming Common Flaws in the Evaluation of Selective Classification Systems
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Imaging（亥姆霍兹成像机构）；University of Heidelberg（海德堡大学）；Heidelberg University Hospital（海德堡大学医院）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）
Position: Graph Condensation Needs a Reset -- Move Beyond Full-dataset Training and Model-Dependence
Yardi School of Artificial Intelligence, IIT Delhi, India（印度德里理工学院Yardi人工智能学院）；Department of Computer Science and Engineering, IIT Delhi, India（印度德里理工学院计算机科学与工程系）；Department of Chemical Engineering, IIT Delhi, India（印度德里理工学院化学工程系）；Indian Institute of Technology Delhi, Abu Dhabi, Zayed City, Abu Dhabi, UAE（印度德里理工学院阿布扎赫德分校，扎耶德城，阿布扎赫德，阿联酋）
