Graph neural networks (GNNs) have achieved great success in many graph learning tasks. The main aspect powering existing GNNs is the multi-layer network architecture to learn the nonlinear graph representations for the specific learning tasks. The core operation in GNNs is message propagation in which each node updates its representation by aggregating its neighbors' representations. Existing GNNs mainly adopt either linear neighborhood aggregation (mean,sum) or max aggregator in their message propagation. (1) For linear aggregators, the whole nonlinearity and network's capacity of GNNs are generally limited due to deeper GNNs usually suffer from over-smoothing issue. (2) For max aggregator, it usually fails to be aware of the detailed information of node representations within neighborhood. To overcome these issues, we re-think the message propagation mechanism in GNNs and aim to develop the general nonlinear aggregators for neighborhood information aggregation in GNNs. One main aspect of our proposed nonlinear aggregators is that they provide the optimally balanced aggregators between max and mean/sum aggregations. Thus, our aggregators can inherit both (i) high nonlinearity that increases network's capacity and (ii) detail-sensitivity that preserves the detailed information of representations together in GNNs' message propagation. Promising experiments on several datasets show the effectiveness of the proposed nonlinear aggregators.
The University of Sydney（悉尼大学）；Stevens Institute of Technology（斯蒂文斯理工学院）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；JD Explore Academy（京东探索学院）
In this paper, we study a novel meta aggregation scheme towards binarizing graph neural networks (GNNs). We begin by developing a vanilla 1-bit GNN framework that binarizes both the GNN parameters and the graph features. Despite the lightweight architecture, we observed that this vanilla framework suffered from insufficient discriminative power in distinguishing graph topologies, leading to a dramatic drop in performance. This discovery motivates us to devise meta aggregators to improve the expressive power of vanilla binarized GNNs, of which the aggregation schemes can be adaptively changed i…
Recently, the Weisfeiler-Lehman (WL) graph isomorphism test was used to measure the expressiveness of graph neural networks (GNNs), showing that the neighborhood aggregation GNNs were at most as powerful as 1-WL test in distinguishing graph structures. There were also improvements proposed in analogy to $k$-WL test ($k>1$). However, the aggregators in these GNNs are far from injective as required by the WL test, and suffer from weak distinguishing strength, making it become expressive bottlenecks. In this paper, we improve the expressiveness by exploring powerful aggregators. We reformulate ag…
The mechanism of message passing in graph neural networks (GNNs) is still mysterious. Apart from convolutional neural networks, no theoretical origin for GNNs has been proposed. To our surprise, message passing can be best understood in terms of power iteration. By fully or partly removing activation functions and layer weights of GNNs, we propose subspace power iteration clustering (SPIC) models that iteratively learn with only one aggregator. Experiments show that our models extend GNNs and enhance their capability to process random featured networks. Moreover, we demonstrate the redundancy …
Graph neural networks (GNNs) have been shown to be highly sensitive to the choice of aggregation function. While summing over a node's neighbours can approximate any permutation-invariant function over discrete inputs, Cohen-Karlik et al. [2020] proved there are set-aggregation problems for which summing cannot generalise to unbounded inputs, proposing recurrent neural networks regularised towards permutation-invariance as a more expressive aggregator. We show that these results carry over to the graph domain: GNNs equipped with recurrent aggregators are competitive with state-of-the-art permu…
POSTECH（浦项科技大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Learning graph-structured data with graph neural networks (GNNs) has been recently emerging as an important field because of its wide applicability in bioinformatics, chemoinformatics, social network analysis and data mining. Recent GNN algorithms are based on neural message passing, which enables GNNs to integrate local structures and node features recursively. However, past GNN algorithms based on 1-hop neighborhood neural message passing are exposed to a risk of loss of information on local structures and relationships. In this paper, we propose Neighborhood Edge AggregatoR (NEAR), a framew…
Graph Neural Networks (GNNs) have been shown to be effective models for different predictive tasks on graph-structured data. Recent work on their expressive power has focused on isomorphism tasks and countable feature spaces. We extend this theoretical framework to include continuous features - which occur regularly in real-world input domains and within the hidden layers of GNNs - and we demonstrate the requirement for multiple aggregation functions in this context. Accordingly, we propose Principal Neighbourhood Aggregation (PNA), a novel architecture combining multiple aggregators with degr…
Graph Neural Networks (GNNs) have led to state-of-the-art performance on a variety of machine learning tasks such as recommendation, node classification and link prediction. Graph neural network models generate node embeddings by merging nodes features with the aggregated neighboring nodes information. Most existing GNN models exploit a single type of aggregator (e.g., mean-pooling) to aggregate neighboring nodes information, and then add or concatenate the output of aggregator to the current representation vector of the center node. However, using only a single type of aggregator is difficult…
Theoretical guarantees for the advantage of GNNs over NNs in generalizing bandlimited functions on Euclidean cubes
Faculty of Mathematics, University of Vienna（维也纳大学数学学院）；Department of Mathematics, Michigan State University（密歇根州立大学数学系）；Department of Statistics and Probability, Michigan State University（密歇根州立大学统计与概率系）
Michigan State University（密歇根州立大学）；New Jersey Institute of Technology（新泽西理工学院）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Recent years have witnessed remarkable success achieved by graph neural networks (GNNs) in many real-world applications such as recommendation and drug discovery. Despite the success, oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. It indicates that the learned node representations are highly indistinguishable due to the stacked aggregators. In this paper, we propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. Through empirical and theoretical study on this matter, we demonstrate the …
École Polytechnique（巴黎综合理工学院）；Institut Polytechnique de Paris（巴黎理工学院）
Graph Neural Networks (GNNs) have been successfully used in many problems involving graph-structured data, achieving state-of-the-art performance. GNNs typically employ a message-passing scheme, in which every node aggregates information from its neighbors using a permutation-invariant aggregation function. Standard well-examined choices such as the mean or sum aggregation functions have limited capabilities, as they are not able to capture interactions among neighbors. In this work, we formalize these interactions using an information-theoretic framework that notably includes synergistic info…
After observing a snapshot of a social network, a link prediction (LP) algorithm identifies node pairs between which new edges will likely materialize in future. Most LP algorithms estimate a score for currently non-neighboring node pairs, and rank them by this score. Recent LP systems compute this score by comparing dense, low dimensional vector representations of nodes. Graph neural networks (GNNs), in particular graph convolutional networks (GCNs), are popular examples. For two nodes to be meaningfully compared, their embeddings should be indifferent to reordering of their neighbors. GNNs t…
The Graduate Center, The City University of New York（纽约市立大学研究生中心）；Hunter College, The City University of New York（纽约市立大学亨特学院）
Graph Neural Networks (GNNs) are powerful to learn the representation of graph-structured data. Most of the GNNs use the message-passing scheme, where the embedding of a node is iteratively updated by aggregating the information of its neighbors. To achieve a better expressive capability of node influences, attention mechanism has grown to be popular to assign trainable weights to the nodes in aggregation. Though the attention-based GNNs have achieved remarkable results in various tasks, a clear understanding of their discriminative capacities is missing. In this work, we present a theoretical…
University of Michigan（密歇根大学）；Northwestern University（西北大学）；RWTH Aachen University（亚琛工业大学）
We bridge two research directions on graph neural networks (GNNs), by formalizing the relation between heterophily of node labels (i.e., connected nodes tend to have dissimilar labels) and the robustness of GNNs to adversarial attacks. Our theoretical and empirical analyses show that for homophilous graph data, impactful structural attacks always lead to reduced homophily, while for heterophilous graph data the change in the homophily level depends on the node degrees. These insights have practical implications for defending against attacks on real-world graphs: we deduce that separate aggrega…
Texas A&M University（德克萨斯农工大学）；The Hong Kong Polytechnic University（香港理工大学）；Samsung Research America（三星美国研究院）
Graph neural networks (GNNs), which learn the representation of a node by aggregating its neighbors, have become an effective computational tool in downstream applications. Over-smoothing is one of the key issues which limit the performance of GNNs as the number of layers increases. It is because the stacked aggregators would make node representations converge to indistinguishable vectors. Several attempts have been made to tackle the issue by bringing linked node pairs close and unlinked pairs distinct. However, they often ignore the intrinsic community structures and would result in sub-opti…
University of Catania（卡塔尼亚大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
Networks are a powerful tool to model complex systems, and the definition of many Graph Neural Networks (GNN), Deep Learning algorithms that can handle networks, has opened a new way to approach many real-world problems that would be hardly or even untractable. In this paper, we propose mGNN, a framework meant to generalize GNNs to the case of multi-layer networks, i.e., networks that can model multiple kinds of interactions and relations between nodes. Our approach is general (i.e., not task specific) and has the advantage of extending any type of GNN without any computational overhead. We te…
SEGNO: Generalizing Equivariant Graph Neural Networks with Physical Inductive Biases
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯AI实验室）
Understanding Oversmoothing in Diffusion-Based GNNs From the Perspective of Operator Semigroup Theory
The School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；The Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Agricultural Information Institute, China Academy of Agricultural Sciences（中国农业科学院农业信息研究所）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Peng Cheng Laboratory（鹏城实验室）
Recurrent Graph Neural Networks and Arithmetic Circuits
Institut für Theoretische Informatik, Leibniz Universität Hannover, Germany（汉诺威莱布尼茨大学理论计算机研究所）；School of Computing Science, University of Glasgow, UK（格拉斯哥大学计算机科学学院）
Distinguishing the automorphic equivalence of nodes in a graph plays an essential role in many scientific domains, e.g., computational biologist and social network analysis. However, existing graph neural networks (GNNs) fail to capture such an important property. To make GNN aware of automorphic equivalence, we first introduce a localized variant of this concept -- ego-centered automorphic equivalence (Ego-AE). Then, we design a novel variant of GNN, i.e., GRAPE, that uses learnable AE-aware aggregators to explicitly differentiate the Ego-AE of each node's neighbors with the aids of various s…
KTH Royal Institute of Technology（皇家理工学院）；University of Peloponnese（伯罗奔尼撒大学）；University of Luxembourg（卢森堡大学）；Ecole Polytechnique, IP Paris（巴黎综合理工学院（巴黎IP））
Large Language Models Meet Graph Neural Networks: A Perspective of Graph Mining
School of Computer Science and Engineering（计算机科学与工程学院）；University of Electronic Science and Technology of China（电子科技大学）；th Research Institute of CETC（中国电子科技集团公司第五十四研究所）
Renmin University of China（中国人民大学）；Alibaba Group（阿里巴巴集团）
Graph Neural Networks (GNNs) have received extensive research attention for their promising performance in graph machine learning. Despite their extraordinary predictive accuracy, existing approaches, such as GCN and GPRGNN, are not robust in the face of homophily changes on test graphs, rendering these models vulnerable to graph structural attacks and with limited capacity in generalizing to graphs of varied homophily levels. Although many methods have been proposed to improve the robustness of GNN models, most of these techniques are restricted to the spatial domain and employ complicated de…
Dalian University of Technology（大连理工大学）；Peng Cheng Laboratory（鹏城实验室）
Many improvements on GNNs can be deemed as operations on the spectrum of the underlying graph matrix, which motivates us to directly study the characteristics of the spectrum and their effects on GNN performance. By generalizing most existing GNN architectures, we show that the correlation issue caused by the $unsmooth$ spectrum becomes the obstacle to leveraging more powerful graph filters as well as developing deep architectures, which therefore restricts GNNs' performance. Inspired by this, we propose the correlation-free architecture which naturally removes the correlation issue among diff…
Weizmann Institute of Science（魏茨曼科学研究所）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；NVIDIA Research（英伟达研究院）；Meta AI Research Centre for Artificial Intelligence（Meta人工智能研究中心）
Aalborg University（奥尔堡大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
Deep neural networks have revolutionized many machine learning tasks in power systems, ranging from pattern recognition to signal processing. The data in these tasks is typically represented in Euclidean domains. Nevertheless, there is an increasing number of applications in power systems, where data are collected from non-Euclidean domains and represented as graph-structured data with high dimensional features and interdependency among nodes. The complexity of graph-structured data has brought significant challenges to the existing deep neural networks defined in Euclidean domains. Recently, …
GRAIN: Multi-Granular and Implicit Information Aggregation Graph Neural Network for Heterophilous Graphs
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
Graph neural networks (GNNs) can process graphs of different sizes, but their ability to generalize across sizes, specifically from small to large graphs, is still not well understood. In this paper, we identify an important type of data where generalization from small to large graphs is challenging: graph distributions for which the local structure depends on the graph size. This effect occurs in multiple important graph learning domains, including social and biological networks. We first prove that when there is a difference between the local structures, GNNs are not guaranteed to generalize…
Existing Graph Neural Networks (GNNs) usually assume a balanced situation where both the class distribution and the node degree distribution are balanced. However, in real-world situations, we often encounter cases where a few classes (i.e., head class) dominate other classes (i.e., tail class) as well as in the node degree perspective, and thus naively applying existing GNNs eventually fall short of generalizing to the tail cases. Although recent studies proposed methods to handle long-tail situations on graphs, they only focus on either the class long-tailedness or the degree long-tailedness…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Graph representation learning is a fundamental research issue and benefits a wide range of applications on graph-structured data. Conventional artificial neural network-based methods such as graph neural networks (GNNs) and variational graph auto-encoders (VGAEs) have achieved promising results in learning on graphs, but they suffer from extremely high energy consumption during training and inference stages. Inspired by the bio-fidelity and energy-efficiency of spiking neural networks (SNNs), recent methods attempt to adapt GNNs to the SNN framework by substituting spiking neurons for the acti…
Ludwig-Maximilian University of Munich（慕尼黑大学）；MILA（蒙特利尔学习算法研究所）；New Jersey Institute of Technology（新泽西理工学院）；TU Wien（维也纳工业大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Vienna（维也纳大学）；NVIDIA Research（英伟达研究院）
Graph neural networks (GNNs) are the primary tool for processing graph-structured data. Unfortunately, the most commonly used GNNs, called Message Passing Neural Networks (MPNNs) suffer from several fundamental limitations. To overcome these limitations, recent works have adapted the idea of positional encodings to graph data. This paper draws inspiration from the recent success of Laplacian-based positional encoding and defines a novel family of positional encoding schemes for graphs. We accomplish this by generalizing the optimization problem that defines the Laplace embedding to more genera…
Semi-decentralized Training of Spatio-Temporal Graph Neural Networks for Traffic Prediction
University of Zagreb, Faculty of Electrical Engineering and Computing（Zagreb大学电气工程与计算学院）；RISE Research Institutes of Sweden（瑞典RISE研究机构）；KTH Royal Institute of Technology（皇家理工学院）
Enhancing Graph Representations with Neighborhood-Contextualized Message-Passing
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Kyoto University（京都大学）；Ateneo de Manila University（马尼拉雅典耀大学）；UNI-President Information Philippines Corporation（统一信息菲律宾公司）；The Chinese University of Hong Kong（香港中文大学）
Ant Financial Services Group（蚂蚁金融服务集团）；Georgia Institute of Technology（佐治亚理工学院）
Several sampling algorithms with variance reduction have been proposed for accelerating the training of Graph Convolution Networks (GCNs). However, due to the intractable computation of optimal sampling distribution, these sampling algorithms are suboptimal for GCNs and are not applicable to more general graph neural networks (GNNs) where the message aggregator contains learned weights rather than fixed weights, such as Graph Attention Networks (GAT). The fundamental reason is that the embeddings of the neighbors or learned weights involved in the optimal sampling distribution are changing dur…
Plain Transformers are Surprisingly Powerful Link Predictors
Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, USA.（计算机科学与工程系，密歇根州立大学，东兰辛，MI，美国。）；Department of Computer Science（计算机科学系）；Engineering, Michigan State University, East Lansing, MI, USA.（工程，密歇根州立大学，东兰辛，MI，美国。）
Simple and Efficient Heterogeneous Graph Neural Network
Institute for Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Griffith University（格里菲斯大学）；University of Chinese Academy of Sciences（中国科学院大学）
Learning representations for graph-structured data is essential for graph analytical tasks. While remarkable progress has been made on static graphs, researches on temporal graphs are still in its beginning stage. The bottleneck of the temporal graph representation learning approach is the neighborhood aggregation strategy, based on which graph attributes share and gather information explicitly. Existing neighborhood aggregation strategies fail to capture either the short-term features or the long-term features of temporal graph attributes, leading to unsatisfactory model performance and even …
Generalizing GNNs with Tokenized Mixture of Experts
University of Connecticut Storrs（康涅狄格大学斯特劳斯分校）；University of Notre Dame（Notre Dame 大学）；University of Virginia（弗吉尼亚大学）；Northwestern University Evanston（北western 大学埃文斯顿分校）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Graph or network data is ubiquitous in the real world, including social networks, information networks, traffic networks, biological networks and various technical networks. The non-Euclidean nature of graph data poses the challenge for modeling and analyzing graph data. Recently, Graph Neural Network (GNNs) are proposed as a general and powerful framework to handle tasks on graph data, e.g., node embedding, link prediction and node classification. As a representative implementation of GNNs, Graph Attention Networks (GATs) are successfully applied in a variety of tasks on real datasets. Howeve…
RWTH Aachen University（亚琛工业大学）；Department of Physics, RWTH Aachen University（物理系，亚琛工业大学）；Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Centre（先进模拟研究所（IAS-6），计算与系统神经科学，耶鲁研究中心）
Institute of Informatics（信息学研究所）；UFRGS（巴西南大河州联邦大学）；Rice University（莱斯大学）
Graph Neural Networks (GNN) are a promising technique for bridging differential programming and combinatorial domains. GNNs employ trainable modules which can be assembled in different configurations that reflect the relational structure of each problem instance. In this paper, we show that GNNs can learn to solve, with very little supervision, the decision variant of the Traveling Salesperson Problem (TSP), a highly relevant $\mathcal{NP}$-Complete problem. Our model is trained to function as an effective message-passing algorithm in which edges (embedded with their weights) communicate with …
