RWTH Aachen University（亚琛工业大学）；Department of Physics, RWTH Aachen University（物理系，亚琛工业大学）；Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Centre（先进模拟研究所（IAS-6），计算与系统神经科学，耶鲁研究中心）
Bastian Epping, Alexandre René, Moritz Helias, Michael T. Schaub
发表信息：
Advances in Neural Information Processing Systems 37 (NeurIPS 2024)
Graph neural networks (GNNs) have emerged as powerful tools for processing relational data in applications. However, GNNs suffer from the problem of oversmoothing, the property that the features of all nodes exponentially converge to the same vector over layers, prohibiting the design of deep GNNs. In this work we study oversmoothing in graph convolutional networks (GCNs) by using their Gaussian process (GP) equivalence in the limit of infinitely many hidden features. By generalizing methods from conventional deep neural networks (DNNs), we can describe the distribution of features at the output layer of deep GCNs in terms of a GP: as expected, we find that typical parameter choices from the literature lead to oversmoothing. The theory, however, allows us to identify a new, non-oversmoothing phase: if the initial weights of the network have sufficiently large variance, GCNs do not oversmooth, and node features remain informative even at large depth. We demonstrate the validity of this prediction in finite-size GCNs by training a linear classifier on their output. Moreover, using the linearization of the GCN GP, we generalize the concept of propagation depth of information from DNNs to GCNs. This propagation depth diverges at the transition between the oversmoothing and non-oversmoothing phase. We test the predictions of our approach and find good agreement with finite-size GCNs. Initializing GCNs near the transition to the non-oversmoothing phase, we obtain networks which are both deep and expressive.
Taming Gradient Oversmoothing and Expansion in Graph Neural Networks
Graduate School of Artificial Intelligence（人工智能研究生院）；Department of Computer Science and Engineering（计算机科学与工程系）；Pohang University of Science and Technology (POSTECH)（浦项科技大学（POSTECH））
Massachusetts Institute of Technology（麻省理工学院）；New York University（纽约大学）
Oversmoothing is a central challenge of building more powerful Graph Neural Networks (GNNs). While previous works have only demonstrated that oversmoothing is inevitable when the number of graph convolutions tends to infinity, in this paper, we precisely characterize the mechanism behind the phenomenon via a non-asymptotic analysis. Specifically, we distinguish between two different effects when applying graph convolutions -- an undesirable mixing effect that homogenizes node representations in different classes, and a desirable denoising effect that homogenizes node representations in the sam…
Are We Measuring Oversmoothing in Graph Neural Networks Correctly?
School of Mathematics and Maxwell Institute University of Edinburgh（数学系和麦克斯韦研究所爱丁堡大学）；Gran Sasso Science Institute（格兰萨索科学研究所）；Scuola Normale Superiore（正常大学）
Demystifying Oversmoothing in Attention-Based Graph Neural Networks
Institute for Data, Systems and Society (IDSS), MIT（麻省理工学院数据、系统与社会研究所）；Laboratory for Information and Decision Systems (LIDS), MIT（麻省理工学院信息与决策系统实验室）；Department of Computing and Mathematical Sciences (CMS), Caltech（加州理工学院计算与数学科学学院）
Reducing Oversmoothing through Informed Weight Initialization in Graph Neural Networks
School of Electrical Engineering and Computer Science, National Technical University of Athens（雅典国立技术大学电气与计算机工程学院）；National Center for Scientific Research “Demokritos”（德谟克利特国家科学研究中心）
Carnegie Mellon University（卡内基梅隆大学）；Pittsburgh, PA 15213, USA（美国宾夕法尼亚州匹兹堡市（15213））
The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers. This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable. We take a closer look at two different interpretations, aiming to quantify oversmoothing. Our main contribution is PairNorm, a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar. What is more, PairNorm is fast, easy to implement without any change t…
Node features of graph neural networks (GNNs) tend to become more similar with the increase of the network depth. This effect is known as over-smoothing, which we axiomatically define as the exponential convergence of suitable similarity measures on the node features. Our definition unifies previous approaches and gives rise to new quantitative measures of over-smoothing. Moreover, we empirically demonstrate this behavior for several over-smoothing measures on different graphs (small-, medium-, and large-scale). We also review several approaches for mitigating over-smoothing and empirically te…
Analyzing the Effect of Embedding Norms and Singular Values to Oversmoothing in Graph Neural Networks
School of Electrical Engineering and Computer Science National Technical University of Athens（电气工程与计算机科学学院国家技术大学雅典）；Archimedes Research Unit, Athena RC（阿基米德研究单位，雅典RC）
Mechanical and Aerospace Engineering（机械与航空航天工程）；Electrical and Computer Engineering（电气与计算机工程）；Center for Statistics and Machine Learning（统计与机器学习中心）；Princeton University（普林斯顿大学）
Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
Graph convolutional networks (GCNs) have shown promising results in processing graph data by extracting structure-aware features. This gave rise to extensive work in geometric deep learning, focusing on designing network architectures that ensure neuron activations conform to regularity patterns within the input graph. However, in most cases the graph structure is only accounted for by considering the similarity of activations between adjacent nodes, which limits the capabilities of such methods to discriminate between nodes in a graph. Here, we propose to augment conventional GCNs with geomet…
Overcoming Oversmoothness in Graph Convolutional Networks via Hybrid Scattering Networks
Mila (Quebec Artificial Intelligence Institute)（米拉（魁北克人工智能研究所））；University of Montreal（蒙特利尔大学）；Cornell University（康奈尔大学）；Michigan State University（密歇根州立大学）；University of California, Los Angeles（加州大学洛杉矶分校）
University of Cambridge（剑桥大学）；Twitter（推特）；University of Oxford（牛津大学）
Cellular sheaves equip graphs with a "geometrical" structure by assigning vector spaces and linear maps to nodes and edges. Graph Neural Networks (GNNs) implicitly assume a graph with a trivial underlying sheaf. This choice is reflected in the structure of the graph Laplacian operator, the properties of the associated diffusion equation, and the characteristics of the convolutional models that discretise this equation. In this paper, we use cellular sheaf theory to show that the underlying geometry of the graph is deeply linked with the performance of GNNs in heterophilic settings and their ov…
Oversmoothing as Representation Degeneracy in Neural Sheaf Diffusion
IUF – Leibniz Research Institute for Environmental Medicine（莱比锡环境医学研究所）；DNTOX GmbH（DNTOX公司）；Bioinformatics Group, Ruhr University Bochum（博德姆鲁尔大学生物信息学小组）；Swiss Centre for Applied Human Toxicology (SCAHT)（瑞士应用人类毒理学中心（SCAHT））
Understanding Oversmoothing in Diffusion-Based GNNs From the Perspective of Operator Semigroup Theory
The School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；The Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
Beyond ReLU: Bifurcation, Oversmoothing, and Topological Priors
Université Paris Saclay, Université Paris Cité, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli（巴黎萨克雷大学、巴黎城市大学、巴黎萨克雷高等师范学院、国家科学研究中心、SSA、国家卫生研究院、Borelli中心）；Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS（社会科学分析与数学中心、EHESS、国家科学研究中心）
Clova AI Research, NAVER Corp.（NAVER公司Clova人工智能研究院）；Naver Shopping, NAVER Corp.（NAVER公司NAVER购物）
Existing approaches for graph neural networks commonly suffer from the oversmoothing issue, regardless of how neighborhoods are aggregated. Most methods also focus on transductive scenarios for fixed graphs, leading to poor generalization for unseen graphs. To address these issues, we propose a new graph neural network that considers both edge-based neighborhood relationships and node-based entity features, i.e. Graph Entities with Step Mixture via random walk (GESM). GESM employs a mixture of various steps through random walk to alleviate the oversmoothing problem, attention to dynamically re…
Michigan State University（密歇根州立大学）；New Jersey Institute of Technology（新泽西理工学院）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Recent years have witnessed remarkable success achieved by graph neural networks (GNNs) in many real-world applications such as recommendation and drug discovery. Despite the success, oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. It indicates that the learned node representations are highly indistinguishable due to the stacked aggregators. In this paper, we propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. Through empirical and theoretical study on this matter, we demonstrate the …
We analyze graph smoothing with \emph{mean aggregation}, where each node successively receives the average of the features of its neighbors. Indeed, it has quickly been observed that Graph Neural Networks (GNNs), which generally follow some variant of Message-Passing (MP) with repeated aggregation, may be subject to the oversmoothing phenomenon: by performing too many rounds of MP, the node features tend to converge to a non-informative limit. In the case of mean aggregation, for connected graphs, the node features become constant across the whole graph. At the other end of the spectrum, it is…
Graph neural networks (GNNs) are able to leverage the structure of graph data by passing messages along the edges of the graph. While this allows GNNs to learn features depending on the graph structure, for certain graph topologies it leads to inefficient information propagation and a problem known as oversquashing. This has recently been linked with the curvature and spectral gap of the graph. On the other hand, adding edges to the message-passing graph can lead to increasingly similar node representations and a problem known as oversmoothing. We propose a computationally efficient algorithm …
Seoul National University（首尔国立大学）；University of Michigan（密歇根大学）；Amazon Web Services（亚马逊云科技）
Heterogeneous graph neural networks (GNNs) achieve strong performance on node classification tasks in a semi-supervised learning setting. However, as in the simpler homogeneous GNN case, message-passing-based heterogeneous GNNs may struggle to balance between resisting the oversmoothing that may occur in deep models, and capturing long-range dependencies of graph structured data. Moreover, the complexity of this trade-off is compounded in the heterogeneous graph case due to the disparate heterophily relationships between nodes of different types. To address these issues, we propose a novel het…
Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；Amazon（亚马逊）；Renmin University of China（中国人民大学）
Despite the recent success of graph neural networks (GNN), common architectures often exhibit significant limitations, including sensitivity to oversmoothing, long-range dependencies, and spurious edges, e.g., as can occur as a result of graph heterophily or adversarial attacks. To at least partially address these issues within a simple transparent framework, we consider a new family of GNN layers designed to mimic and integrate the update rules of two classical iterative algorithms, namely, proximal gradient descent and iterative reweighted least squares (IRLS). The former defines an extensib…
While Graph Neural Networks (GNNs) are powerful models for learning representations on graphs, most state-of-the-art models do not have significant accuracy gain beyond two to three layers. Deep GNNs fundamentally need to address: 1). expressivity challenge due to oversmoothing, and 2). computation challenge due to neighborhood explosion. We propose a simple "deep GNN, shallow sampler" design principle to improve both the GNN accuracy and efficiency -- to generate representation of a target node, we use a deep GNN to pass messages only within a shallow, localized subgraph. A properly sampled s…
State-of-the-art Graph Neural Networks (GNNs) have limited scalability with respect to the graph and model sizes. On large graphs, increasing the model depth often means exponential expansion of the scope (i.e., receptive field). Beyond just a few layers, two fundamental challenges emerge: 1. degraded expressivity due to oversmoothing, and 2. expensive computation due to neighborhood explosion. We propose a design principle to decouple the depth and scope of GNNs -- to generate representation of a target entity (i.e., a node or an edge), we first extract a localized subgraph as the bounded-siz…
This paper studies asynchronous message passing (AMP), a new paradigm for applying neural network based learning to graphs. Existing graph neural networks use the synchronous distributed computing model and aggregate their neighbors in each round, which causes problems such as oversmoothing and limits their expressiveness. On the other hand, AMP is based on the asynchronous model, where nodes react to messages of their neighbors individually. We prove that (i) AMP can simulate synchronous GNNs and that (ii) AMP can theoretically distinguish any pair of graphs. We experimentally validate AMP's …
Twitter Inc.（推特公司）；Imperial College London（帝国理工学院）；IDSIA/USI（瑞士人工智能研究所/瑞士意大利语大学）
We present Graph Neural Diffusion (GRAND) that approaches deep learning on graphs as a continuous diffusion process and treats Graph Neural Networks (GNNs) as discretisations of an underlying PDE. In our model, the layer structure and topology correspond to the discretisation choices of temporal and spatial operators. Our approach allows a principled development of a broad new class of GNNs that are able to address the common plights of graph learning models such as depth, oversmoothing, and bottlenecks. Key to the success of our models are stability with respect to perturbations in the data a…
Sloan Kettering Institute（斯隆凯特琳研究所）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；City College of New York（纽约城市学院）；City University of New York（纽约城市大学）；Facebook, Inc.（脸书公司）
Graph neural networks (GNNs) manifest pathologies including over-smoothing and limited discriminating power as a result of suboptimally expressive aggregating mechanisms. We herein present a unifying framework for stochastic aggregation (STAG) in GNNs, where noise is (adaptively) injected into the aggregation process from the neighborhood to form node embeddings. We provide theoretical arguments that STAG models, with little overhead, remedy both of the aforementioned problems. In addition to fixed-noise models, we also propose probabilistic versions of STAG models and a variational inference …
ETH Zürich（苏黎世联邦理工学院）；Twitter Inc.（推特公司）；University of Oxford（牛津大学）
We propose Graph-Coupled Oscillator Networks (GraphCON), a novel framework for deep learning on graphs. It is based on discretizations of a second-order system of ordinary differential equations (ODEs), which model a network of nonlinear controlled and damped oscillators, coupled via the adjacency structure of the underlying graph. The flexibility of our framework permits any basic GNN layer (e.g. convolutional or attentional) as the coupling function, from which a multi-layer deep neural network is built up via the dynamics of the proposed ODEs. We relate the oversmoothing problem, commonly e…
Imperial College London（帝国理工学院）；Ludwig-Maximilians Universität München（慕尼黑大学）；Adobe（奥多比公司）
With the increasing use of Graph Neural Networks (GNNs) in critical real-world applications, several post hoc explanation methods have been proposed to understand their predictions. However, there has been no work in generating explanations on the fly during model training and utilizing them to improve the expressive power of the underlying GNN models. In this work, we introduce a novel explanation-directed neural message passing framework for GNNs, EXPASS (EXplainable message PASSing), which aggregates only embeddings from nodes and edges identified as important by a GNN explanation method. E…
Graph neural networks (GNNs) have achieved outstanding performance in learning graph-structured data and various tasks. However, many current GNNs suffer from three common problems when facing large-size graphs or using a deeper structure: neighbors explosion, node dependence, and oversmoothing. Such problems attribute to the data structures of the graph itself or the designing of the multi-layers GNNs framework, and can lead to low training efficiency and high space complexity. To deal with these problems, in this paper, we propose a general subgraph-based training framework, namely Ripple Wa…
Graph convolutional networks are a popular class of deep neural network algorithms which have shown success in a number of relational learning tasks. Despite their success, graph convolutional networks exhibit a number of peculiar features, including a bias towards learning oversmoothed and homophilic functions, which are not easily diagnosed due to the complex nature of these algorithms. We propose to bridge this gap in understanding by studying the neural tangent kernel of sheaf convolutional networks--a topological generalization of graph convolutional networks. To this end, we derive a par…
Clova AI Research, NAVER Corp.（NAVER公司Clova人工智能研究院）；Search Solution Inc.（搜索解决方案公司）；LINE Plus Corp.（LINE Plus公司）
Graph Neural Networks (GNNs) have been emerging as a promising method for relational representation including recommender systems. However, various challenging issues of social graphs hinder the practical usage of GNNs for social recommendation, such as their complex noisy connections and high heterogeneity. The oversmoothing of GNNs is an obstacle of GNN-based social recommendation as well. Here we propose a new graph embedding method Heterogeneous Graph Propagation (HGP) to tackle these issues. HGP uses a group-user-item tripartite graph as input to reduce the number of edges and the complex…
