Graph Neural Networks (GNNs) play a pivotal role in graph-based tasks for their proficiency in representation learning. Among the various GNN methods, spectral GNNs employing polynomial filters have shown promising performance on tasks involving both homophilous and heterophilous graph structures. However, The scalability of spectral GNNs on large graphs is limited because they learn the polynomial coefficients through multiple forward propagation executions during forward propagation. Existing works have attempted to scale up spectral GNNs by eliminating the linear layers on the input node features, a change that can disrupt end-to-end training, potentially impact performance, and become impractical with high-dimensional input features. To address the above challenges, we propose "Spectral Graph Neural Networks with Laplacian Sparsification (SGNN-LS)", a novel graph spectral sparsification method to approximate the propagation patterns of spectral GNNs. We prove that our proposed method generates Laplacian sparsifiers that can approximate both fixed and learnable polynomial filters with theoretical guarantees. Our method allows the application of linear layers on the input node features, enabling end-to-end training as well as the handling of raw text features. We conduct an extensive experimental analysis on datasets spanning various graph scales and properties to demonstrate the superior efficiency and effectiveness of our method. The results show that our method yields superior results in comparison with the corresponding approximated base models, especially on dataset Ogbn-papers100M(111M nodes, 1.6B edges) and MAG-scholar-C (2.8M features).
Mitigating Over-Squashing in Graph Neural Networks by Spectrum-Preserving Sparsification
Kim Jaechul Graduate School of Artificial Intelligence, KAIST, Seoul, Republic of Korea（韩国釜山科学技术院人工智能研究生院）；School of Electrical Engineering, KAIST, Daejeon, Republic of Korea（韩国釜山科学技术院电子工程学院）；Department of Engineering, University of Cambridge, Cambridge, United Kingdom（剑桥大学工程系）；Incubation Institute, Fudan University, Shanghai, China（复旦大学孵化院）；Shanghai Academy of Artificial Intelligence for Science, Shanghai, China（上海人工智能科学研究院）；School of Information and Communication Technology, Griffith University, Gold Coast, Australia（格里菲斯大学信息与通信技术学院）
Improved large-scale graph learning through ridge spectral sparsification
SequeL team, INRIA Lille - Nord Europe, France（SequeL团队，INRIA里尔-北欧，法国）；Facebook AI Research, Paris, France（Facebook人工智能研究，巴黎，法国）；MIT, USA（麻省理工学院，美国）；New Jersey Institute of Technology, USA（新泽西理工学院，美国）
UiT - The Arctic University of Norway（特罗姆瑟大学-挪威北极大学）；NEC Laboratories Europe（NEC欧洲实验室）；University of Amsterdam（阿姆斯特丹大学）；Norwegian Computing Center（挪威计算中心）；University of Copenhagen（哥本哈根大学）；University of Florida（佛罗里达大学）
Graph sparsification aims to reduce the number of edges of a graph while maintaining its structural properties. In this paper, we propose the first general and effective information-theoretic formulation of graph sparsification, by taking inspiration from the Principle of Relevant Information (PRI). To this end, we extend the PRI from a standard scalar random variable setting to structured data (i.e., graphs). Our Graph-PRI objective is achieved by operating on the graph Laplacian, made possible by expressing the graph Laplacian of a subgraph in terms of a sparse edge selection vector $\mathbf…
The eigendeomposition of nearest-neighbor (NN) graph Laplacian matrices is the main computational bottleneck in spectral clustering. In this work, we introduce a highly-scalable, spectrum-preserving graph sparsification algorithm that enables to build ultra-sparse NN (u-NN) graphs with guaranteed preservation of the original graph spectrums, such as the first few eigenvectors of the original graph Laplacian. Our approach can immediately lead to scalable spectral clustering of large data networks without sacrificing solution quality. The proposed method starts from constructing low-stretch span…
Stevens Institute of Technology（史蒂文斯理工学院）；Michigan Technological University（密歇根理工大学）
Recent spectral graph sparsification techniques have shown promising performance in accelerating many numerical and graph algorithms, such as iterative methods for solving large sparse matrices, spectral partitioning of undirected graphs, vectorless verification of power/thermal grids, representation learning of large graphs, etc. However, prior spectral graph sparsification methods rely on fast Laplacian matrix solvers that are usually challenging to implement in practice. This work, for the first time, introduces a solver-free approach (SF-GRASS) for spectral graph sparsification by leveragi…
INRIA Lille - Nord Europe（法国国家信息与自动化研究所里尔-北欧洲分所）；University of Puerto Rico - Rio Piedras（波多黎各大学里约彼德拉斯分校）
While the harmonic function solution performs well in many semi-supervised learning (SSL) tasks, it is known to scale poorly with the number of samples. Recent successful and scalable methods, such as the eigenfunction method focus on efficiently approximating the whole spectrum of the graph Laplacian constructed from the data. This is in contrast to various subsampling and quantization methods proposed in the past, which may fail in preserving the graph spectra. However, the impact of the approximation of the spectrum on the final generalization error is either unknown, or requires strong ass…
Microsoft Research（微软研究院）；The University of Bristol（布里斯托大学）
For any undirected and weighted graph $G=(V,E,w)$ with $n$ vertices and $m$ edges, we call a sparse subgraph $H$ of $G$, with proper reweighting of the edges, a $(1+\varepsilon)$-spectral sparsifier if \[ (1-\varepsilon)x^{\intercal}L_Gx\leq x^{\intercal} L_{H} x\leq (1+\varepsilon) x^{\intercal} L_Gx \] holds for any $x\in\mathbb{R}^n$, where $L_G$ and $L_{H}$ are the respective Laplacian matrices of $G$ and $H$. Noticing that $Ω(m)$ time is needed for any algorithm to construct a spectral sparsifier and a spectral sparsifier of $G$ requires $Ω(n)$ edges, a natural question is to investigate,…
We apply concepts from manifold regularization to develop new regularization techniques for training locally stable deep neural networks. Our regularizers are based on a sparsification of the graph Laplacian which holds with high probability when the data is sparse in high dimensions, as is common in deep learning. Empirically, our networks exhibit stability in a diverse set of perturbation models, including $\ell_2$, $\ell_\infty$, and Wasserstein-based perturbations; in particular, we achieve 40% adversarial accuracy on CIFAR-10 against an adaptive PGD attack using $\ell_\infty$ perturbation…
University of Wisconsin（威斯康星大学）；North Carolina State University（北卡罗来纳州立大学）
This paper investigates the use of methods from partial differential equations and the Calculus of variations to study learning problems that are regularized using graph Laplacians. Graph Laplacians are a powerful, flexible method for capturing local and global geometry in many classes of learning problems, and the techniques developed in this paper help to broaden the methodology of studying such problems. In particular, we develop the use of maximum principle arguments to establish asymptotic consistency guarantees within the context of noise corrupted, non-parametric regression with samples…
Ecole Polytechnique F\'ed\'erale de Lausanne (EPFL), LTS4, Lausanne, Switzerland
We are interested in multilayer graph clustering, which aims at dividing the graph nodes into categories or communities. To do so, we propose to learn a clustering-friendly embedding of the graph nodes by solving an optimization problem that involves a fidelity term to the layers of a given multilayer graph, and a regularization on the (single-layer) graph induced by the embedding. The fidelity term uses the contrastive loss to properly aggregate the observed layers into a representative embedding. The regularization pushes for a sparse and community-aware graph, and it is based on a measure o…
Michigan Technological University（密歇根理工大学）；Stevens Institute of Technology（史蒂文斯理工学院）
Learning meaningful graphs from data plays important roles in many data mining and machine learning tasks, such as data representation and analysis, dimension reduction, data clustering, and visualization, etc. In this work, for the first time, we present a highly-scalable spectral approach (GRASPEL) for learning large graphs from data. By limiting the precision matrix to be a graph Laplacian, our approach aims to estimate ultra-sparse (tree-like) weighted undirected graphs and shows a clear connection with the prior graphical Lasso method. By interleaving the latest high-performance nearly-li…
University of California, Riverside（加州大学河滨分校）；University of British Columbia（不列颠哥伦比亚大学）
Deep networks are typically trained with many more parameters than the size of the training dataset. Recent empirical evidence indicates that the practice of overparameterization not only benefits training large models, but also assists - perhaps counterintuitively - building lightweight models. Specifically, it suggests that overparameterization benefits model pruning / sparsification. This paper sheds light on these empirical findings by theoretically characterizing the high-dimensional asymptotics of model pruning in the overparameterized regime. The theory presented addresses the following…
