New York University（纽约大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Zhengdao Chen, Lei Chen, Soledad Villar, Joan Bruna
备注：
Improved the descriptions of the Local Relational Pooling (LRP) model and its practical implementation; Added more experimental results on synthetic and molecular datasets; Added the LRP-l-1 model in the experiments
发表信息：
34th Conference on Neural Information Processing Systems (NeurIPS 2020)
The ability to detect and count certain substructures in graphs is important for solving many tasks on graph-structured data, especially in the contexts of computational chemistry and biology as well as social network analysis. Inspired by this, we propose to study the expressive power of graph neural networks (GNNs) via their ability to count attributed graph substructures, extending recent works that examine their power in graph isomorphism testing and function approximation. We distinguish between two types of substructure counting: induced-subgraph-count and subgraph-count, and establish both positive and negative answers for popular GNN architectures. Specifically, we prove that Message Passing Neural Networks (MPNNs), 2-Weisfeiler-Lehman (2-WL) and 2-Invariant Graph Networks (2-IGNs) cannot perform induced-subgraph-count of substructures consisting of 3 or more nodes, while they can perform subgraph-count of star-shaped substructures. As an intermediary step, we prove that 2-WL and 2-IGNs are equivalent in distinguishing non-isomorphic graphs, partly answering an open problem raised in Maron et al. (2019). We also prove positive results for k-WL and k-IGNs as well as negative results for k-WL with a finite number of iterations. We then conduct experiments that support the theoretical results for MPNNs and 2-IGNs. Moreover, motivated by substructure counting and inspired by Murphy et al. (2019), we propose the Local Relational Pooling model and demonstrate that it is not only effective for substructure counting but also able to achieve competitive performance on molecular prediction tasks.
An Efficient Subgraph GNN with Provable Substructure Counting Power
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
At the CERN LHC, the task of jet tagging, whose goal is to infer the origin of a jet given a set of final-state particles, is dominated by machine learning methods. Graph neural networks have been used to address this task by treating jets as point clouds with underlying, learnable, edge connections between the particles inside. We explore the decision-making process for one such state-of-the-art network, ParticleNet, by looking for relevant edge connections identified using the layerwise-relevance propagation technique. As the model is trained, we observe changes in the distribution of releva…
Dalian University of Technology（大连理工大学）；Peng Cheng Laboratory（鹏城实验室）
For learning graph representations, not all detailed structures within a graph are relevant to the given graph tasks. Task-relevant structures can be $localized$ or $sparse$ which are only involved in subgraphs or characterized by the interactions of subgraphs (a hierarchical perspective). A graph neural network should be able to efficiently extract task-relevant structures and be invariant to irrelevant parts, which is challenging for general message passing GNNs. In this work, we propose to learn graph representations from a sequence of subgraphs of the original graph to better capture task-…
Peking University（北京大学）；The Hong Kong University of Science and Technology（香港科技大学）
Graph neural networks (GNNs) have achieved tremendous success in graph mining. However, the inability of GNNs to model substructures in graphs remains a significant drawback. Specifically, message-passing GNNs (MPGNNs), as the prevailing type of GNNs, have been theoretically shown unable to distinguish, detect or count many graph substructures. While efforts have been paid to complement the inability, existing works either rely on pre-defined substructure sets, thus being less flexible, or are lacking in theoretical insights. In this paper, we propose GSKN, a GNN model with a theoretically str…
Beijing University of Posts and Telecommunications（北京邮电大学）；Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；HEC Montréal（蒙特利尔高等商学院）
Most Graph Neural Networks (GNNs) predict the labels of unseen graphs by learning the correlation between the input graphs and labels. However, by presenting a graph classification investigation on the training graphs with severe bias, surprisingly, we discover that GNNs always tend to explore the spurious correlations to make decision, even if the causal correlation always exists. This implies that existing GNNs trained on such biased datasets will suffer from poor generalization capability. By analyzing this problem in a causal view, we find that disentangling and decorrelating the causal an…
Peking University（北京大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Central Florida（中佛罗里达大学）
Point-of-Interest (POI) recommendation, which benefits from the proliferation of GPS-enabled devices and location-based social networks (LBSNs), plays an increasingly important role in recommender systems. It aims to provide users with the convenience to discover their interested places to visit based on previous visits and current status. Most existing methods usually merely leverage recurrent neural networks (RNNs) to explore sequential influences for recommendation. Despite the effectiveness, these methods not only neglect topological geographical influences among POIs, but also fail to mod…
Graph Neural Networks (GNNs) are widely used for graph representation learning in many application domains. The expressiveness of vanilla GNNs is upper-bounded by 1-dimensional Weisfeiler-Leman (1-WL) test as they operate on rooted subtrees through iterative message passing. In this paper, we empower GNNs by injecting neighbor-connectivity information extracted from a new type of substructure. We first investigate different kinds of connectivities existing in a local neighborhood and identify a substructure called union subgraph, which is able to capture the complete picture of the 1-hop neigh…
University of Toronto（多伦多大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；CIFAR（加拿大高级研究院）；Vector Institute（向量研究所）
Graph neural network have achieved impressive results in predicting molecular properties, but they do not directly account for local and hidden structures in the graph such as functional groups and molecular geometry. At each propagation step, GNNs aggregate only over first order neighbours, ignoring important information contained in subsequent neighbours as well as the relationships between those higher order connections. In this work, we generalize graph neural nets to pass messages and aggregate across higher order paths. This allows for information to propagate over various levels and sub…
Despite that going deep has proven successful in many neural architectures, the existing graph transformers are relatively shallow. In this work, we explore whether more layers are beneficial to graph transformers, and find that current graph transformers suffer from the bottleneck of improving performance by increasing depth. Our further analysis reveals the reason is that deep graph transformers are limited by the vanishing capacity of global attention, restricting the graph transformer from focusing on the critical substructure and obtaining expressive features. To this end, we propose a no…
Building explainable graph neural network by sparse learning for the drug-protein binding prediction
Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington（印第安纳大学伯明顿分校勒迪信息学、计算与工程学院）；School of Medicine, Indiana University（印第安纳大学医学院）
Graph Neural Networks (GNNs) have enjoyed wide spread applications in graph-structured data. However, existing graph based applications commonly lack annotated data. GNNs are required to learn latent patterns from a limited amount of training data to perform inferences on a vast amount of test data. The increased complexity of GNNs, as well as a single point of model parameter initialization, usually lead to overfitting and sub-optimal performance. In addition, it is known that GNNs are vulnerable to adversarial attacks. In this paper, we push one step forward on the ensemble learning of GNNs …
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Life Sciences, Northwestern Polytechnical University（西北工业大学生命学院）
Motivation: Computational prediction of multiple-type drug-drug interaction (DDI) helps reduce unexpected side effects in poly-drug treatments. Although existing computational approaches achieve inspiring results, they ignore that the action of a drug is mainly caused by its chemical substructures. In addition, their interpretability is still weak. Results: In this paper, by supposing that the interactions between two given drugs are caused by their local chemical structures (sub-structures) and their DDI types are determined by the linkages between different substructure sets, we design a nov…
As Graph Neural Networks (GNNs) are widely adopted in digital pathology, there is increasing attention to developing explanation models (explainers) of GNNs for improved transparency in clinical decisions. Existing explainers discover an explanatory subgraph relevant to the prediction. However, such a subgraph is insufficient to reveal all the critical biological substructures for the prediction because the prediction will remain unchanged after removing that subgraph. Hence, an explanatory subgraph should be not only necessary for prediction, but also sufficient to uncover the most predictive…
University of Bologna（博洛尼亚大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
We propose a novel architecture for Graph Neural Networks that is inspired by the idea behind Tree Kernels of measuring similarity between trees by taking into account their common substructures, named fragments. By imposing a series of regularization constraints to the learning problem, we exploit a pooling mechanism that incorporates such notion of fragments within the node soft assignment function that produces the embeddings. We present an extensive experimental evaluation on a collection of sentence classification tasks conducted on several argument mining corpora, showing that the propos…
NGTM: Substructure-based Neural Graph Topic Model for Interpretable Graph Generation
The Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（人工智能前沿技术研究所，香港科学与技术大学（广州））；The College of Computer Science and Technology, The Nanjing University of Aeronautics and Astronautics（计算机科学与技术学院，南京航空航天大学）
IBM Research（IBM研究院）；Carnegie Mellon University（卡内基梅隆大学）；George Mason University（乔治梅森大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Zhejiang University（浙江大学）
Analysis of large-scale sequential data has been one of the most crucial tasks in areas such as bioinformatics, text, and audio mining. Existing string kernels, however, either (i) rely on local features of short substructures in the string, which hardly capture long discriminative patterns, (ii) sum over too many substructures, such as all possible subsequences, which leads to diagonal dominance of the kernel matrix, or (iii) rely on non-positive-definite similarity measures derived from the edit distance. Furthermore, while there have been works addressing the computational challenge with re…
Pohang University of Science and Technology (POSTECH)（浦项科技大学）
Designing a neural network architecture for molecular representation is crucial for AI-driven drug discovery and molecule design. In this work, we propose a new framework for molecular representation learning. Our contribution is threefold: (a) demonstrating the usefulness of incorporating substructures to node-wise features from molecules, (b) designing two branch networks consisting of a transformer and a graph neural network so that the networks fused with asymmetric attention, and (c) not requiring heuristic features and computationally-expensive information from molecules. Using 1.8 milli…
Yunnan University（云南大学）；Microsoft Research Asia（微软亚洲研究院）；Chinese Academy of Science（中国科学院）
Based on basis path set, G-SGD algorithm significantly outperforms conventional SGD algorithm in optimizing neural networks. However, how the inner mechanism of basis paths work remains mysterious. From the aspect of graph theory, this paper defines basis path, investigates structure properties of basis paths in regular fully connected neural network and interprets the graph representation of basis path set. Moreover, we propose hierarchical algorithm HBPS to find basis path set B in fully connected neural network by decomposing the network into several independent and parallel substructures. …
University of California, Santa Barbara（加州大学圣塔芭芭拉分校）
Graph-structured data arise in many scenarios. A fundamental problem is to quantify the similarities of graphs for tasks such as classification. R-convolution graph kernels are positive-semidefinite functions that decompose graphs into substructures and compare them. One problem in the effective implementation of this idea is that the substructures are not independent, which leads to high-dimensional feature space. In addition, graph kernels cannot capture the high-order complex interactions between vertices. To mitigate these two problems, we propose a framework called DeepMap to learn deep r…
Adaptive Substructure-Aware Expert Model for Molecular Property Prediction
Institute of Cyberspace Security, College of Information Engineering（信息工程学院网络空间安全研究所）；Binjiang Institute of Artificial Intelligence（滨江人工智能研究所）；Zhejiang University of Technology（浙江工业大学）
The NSF AI Institute for Artificial Intelligence and Fundamental Interactions（NSF人工智能与基础交互AI研究所）；Center for Theoretical Physics, Massachusetts Institute of Technology（麻省理工学院理论物理中心）；Department of Physics, Massachusetts Institute of Technology（麻省理工学院物理系）；Department of Physics, Harvard University（哈佛大学物理系）；Center for Cosmology and Particle Physics, Department of Physics, New York University（纽约大学物理系宇宙学与粒子物理中心）
Astrometry -- the precise measurement of positions and motions of celestial objects -- has emerged as a promising avenue for characterizing the dark matter population in our Galaxy. By leveraging recent advances in simulation-based inference and neural network architectures, we introduce a novel method to search for global dark matter-induced gravitational lensing signatures in astrometric datasets. Our method based on neural likelihood-ratio estimation shows significantly enhanced sensitivity to a cold dark matter population and more favorable scaling with measurement noise compared to existi…
Functional groups (FGs) are molecular substructures that are served as a foundation for analyzing and predicting chemical properties of molecules. Automatic discovery of FGs will impact various fields of research, including medicinal chemistry and material sciences, by reducing the amount of lab experiments required for discovery or synthesis of new molecules. In this paper, we investigate methods based on graph convolutional neural networks (GCNNs) for localizing FGs that contribute to specific chemical properties of interest. In our framework, molecules are modeled as undirected relational g…
Visual question answering is fundamentally compositional in nature---a question like "where is the dog?" shares substructure with questions like "what color is the dog?" and "where is the cat?" This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning *neural module networks*, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and…
Simon Fraser University（西蒙弗雷泽大学）；National University of Defense Technology（国防科技大学）；Princeton University（普林斯顿大学）；Adobe Research（Adobe研究院）；IIT Bombay（印度理工学院孟买分校）
We introduce SCORES, a recursive neural network for shape composition. Our network takes as input sets of parts from two or more source 3D shapes and a rough initial placement of the parts. It outputs an optimized part structure for the composed shape, leading to high-quality geometry construction. A unique feature of our composition network is that it is not merely learning how to connect parts. Our goal is to produce a coherent and plausible 3D shape, despite large incompatibilities among the input parts. The network may significantly alter the geometry and structure of the input parts and s…
De Montfort University（德蒙福特大学）；Universidade Federal do Rio de Janeiro（里约热内卢联邦大学）
This paper presents a method to identify substructures in NMR spectra of mixtures, specifically 2D spectra, using a bespoke image-based Convolutional Neural Network application. This is done using HSQC and HMBC spectra separately and in combination. The application can reliably detect substructures in pure compounds, using a simple network. It can work for mixtures when trained on pure compounds only. HMBC data and the combination of HMBC and HSQC show better results than HSQC alone.
Metal-organic framework (MOFs) are nanoporous materials that could be used to capture carbon dioxide from the exhaust gas of fossil fuel power plants to mitigate climate change. In this work, we design and train a message passing neural network (MPNN) to predict simulated CO$_2$ adsorption in MOFs. Towards providing insights into what substructures of the MOFs are important for the prediction, we introduce a soft attention mechanism into the readout function that quantifies the contributions of the node representations towards the graph representations. We investigate different mechanisms for …
