This work aims at improving the energy efficiency of decentralized learning by optimizing the mixing matrix, which controls the communication demands during the learning process. Through rigorous analysis based on a state-of-the-art decentralized learning algorithm, the problem is formulated as a bi-level optimization, with the lower level solved by graph sparsification. A solution with guaranteed performance is proposed for the special case of fully-connected base topology and a greedy heuristic is proposed for the general case. Simulations based on real topology and dataset show that the proposed solution can lower the energy consumption at the busiest node by 54%-76% while maintaining the quality of the trained model.
Distributed learning techniques such as federated learning have enabled multiple workers to train machine learning models together to reduce the overall training time. However, current distributed training algorithms (centralized or decentralized) suffer from the communication bottleneck on multiple low-bandwidth workers (also on the server under the centralized architecture). Although decentralized algorithms generally have lower communication complexity than the centralized counterpart, they still suffer from the communication bottleneck for workers with low network bandwidth. To deal with t…
University of North Carolina Greensboro（北卡罗来纳大学格林斯伯勒分校）；University of Connecticut（康涅狄格大学）
Graph sparsification is a powerful tool to approximate an arbitrary graph and has been used in machine learning over homogeneous graphs. In heterogeneous graphs such as knowledge graphs, however, sparsification has not been systematically exploited to improve efficiency of learning tasks. In this work, we initiate the study on heterogeneous graph sparsification and develop sampling-based algorithms for constructing sparsifiers that are provably sparse and preserve important information in the original graphs. We have performed extensive experiments to confirm that the proposed method can impro…
The interconnectedness and interdependence of modern graphs are growing ever more complex, causing enormous resources for processing, storage, communication, and decision-making of these graphs. In this work, we focus on the task graph sparsification: an edge-reduced graph of a similar structure to the original graph is produced while various user-defined graph metrics are largely preserved. Existing graph sparsification methods are mostly sampling-based, which introduce high computation complexity in general and lack of flexibility for a different reduction objective. We present SparRL, the f…
We study epidemic forecasting on real-world health data by a graph-structured recurrent neural network (GSRNN). We achieve state-of-the-art forecasting accuracy on the benchmark CDC dataset. To improve model efficiency, we sparsify the network weights via transformed-$\ell_1$ penalty and maintain prediction accuracy at the same level with 70% of the network weights being zero.
Graph Neural Networks (GNNs) are widely used to perform different machine learning tasks on graphs. As the size of the graphs grows, and the GNNs get deeper, training and inference time become costly in addition to the memory requirement. Thus, without sacrificing accuracy, graph sparsification, or model compression becomes a viable approach for graph learning tasks. A few existing techniques only study the sparsification of graphs and GNN models. In this paper, we develop a SparseGCN pipeline to study all possible sparsification in GNN. We provide a theoretical analysis and empirically show t…
Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsification
Department of Electrical, Computer, and Systems Engineering（电气、计算机与系统工程系）；Rensselaer Polytechnic Institute（伦塞拉尔理工大学）；Department of Electrical and Systems Engineering（电气与系统工程系）；University of Pennsylvania（宾夕法尼亚大学）
Scaffold: Support Graph Theory Based Sparsification for Graph Neural Networks
Pacific Northwest National Laboratory（太平洋西北国家实验室）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Adobe（Adobe公司）；Boise State University（博伊西州立大学）；Purdue University（普渡大学）
Shapley-Value-Based Graph Sparsification for GNN Inference
Department of Intelligent Systems Engineering Indiana University Bloomington Bloomington Indiana USA；Department of Computer Science \& Engineering Texas A\&M University College Station Texas USA；Indiana University Bloomington；Texas A\&M University
UiT - The Arctic University of Norway（特罗姆瑟大学-挪威北极大学）；NEC Laboratories Europe（NEC欧洲实验室）；University of Amsterdam（阿姆斯特丹大学）；Norwegian Computing Center（挪威计算中心）；University of Copenhagen（哥本哈根大学）；University of Florida（佛罗里达大学）
Graph sparsification aims to reduce the number of edges of a graph while maintaining its structural properties. In this paper, we propose the first general and effective information-theoretic formulation of graph sparsification, by taking inspiration from the Principle of Relevant Information (PRI). To this end, we extend the PRI from a standard scalar random variable setting to structured data (i.e., graphs). Our Graph-PRI objective is achieved by operating on the graph Laplacian, made possible by expressing the graph Laplacian of a subgraph in terms of a sparse edge selection vector $\mathbf…
Georgia Institute of Technology（佐治亚理工学院）；IQVIA（艾昆纬）；University of Illinois, Urbana-Champagne（伊利诺伊大学厄巴纳-香槟分校）
The attention mechanism has demonstrated superior performance for inference over nodes in graph neural networks (GNNs), however, they result in a high computational burden during both training and inference. We propose FastGAT, a method to make attention based GNNs lightweight by using spectral sparsification to generate an optimal pruning of the input graph. This results in a per-epoch time that is almost linear in the number of graph nodes as opposed to quadratic. We theoretically prove that spectral sparsification preserves the features computed by the GAT model, thereby justifying our algo…
We derive a new proof to show that the incremental resparsification algorithm proposed by Kelner and Levin (2013) produces a spectral sparsifier in high probability. We rigorously take into account the dependencies across subsequent resparsifications using martingale inequalities, fixing a flaw in the original analysis.
We propose an end-to-end architecture for multivariate time-series prediction that integrates a spatial-temporal graph neural network with a matrix filtering module. This module generates filtered (inverse) correlation graphs from multivariate time series before inputting them into a GNN. In contrast with existing sparsification methods adopted in graph neural network, our model explicitly leverage time-series filtering to overcome the low signal-to-noise ratio typical of complex systems data. We present a set of experiments, where we predict future sales from a synthetic time-series sales dat…
Stevens Institute of Technology（史蒂文斯理工学院）；Michigan Technological University（密歇根理工大学）
Recent spectral graph sparsification techniques have shown promising performance in accelerating many numerical and graph algorithms, such as iterative methods for solving large sparse matrices, spectral partitioning of undirected graphs, vectorless verification of power/thermal grids, representation learning of large graphs, etc. However, prior spectral graph sparsification methods rely on fast Laplacian matrix solvers that are usually challenging to implement in practice. This work, for the first time, introduces a solver-free approach (SF-GRASS) for spectral graph sparsification by leveragi…
INRIA Lille - Nord Europe（法国国家信息与自动化研究所里尔-北欧洲分所）；University of Puerto Rico - Rio Piedras（波多黎各大学里约彼德拉斯分校）
While the harmonic function solution performs well in many semi-supervised learning (SSL) tasks, it is known to scale poorly with the number of samples. Recent successful and scalable methods, such as the eigenfunction method focus on efficiently approximating the whole spectrum of the graph Laplacian constructed from the data. This is in contrast to various subsampling and quantization methods proposed in the past, which may fail in preserving the graph spectra. However, the impact of the approximation of the spectrum on the final generalization error is either unknown, or requires strong ass…
University of Connecticut（康涅狄格大学）；University of Rochester（罗切斯特大学）；George Mason University（乔治梅森大学）
As real-world graphs expand in size, larger GNN models with billions of parameters are deployed. High parameter count in such models makes training and inference on graphs expensive and challenging. To reduce the computational and memory costs of GNNs, optimization methods such as pruning the redundant nodes and edges in input graphs have been commonly adopted. However, model compression, which directly targets the sparsification of model layers, has been mostly limited to traditional Deep Neural Networks (DNNs) used for tasks such as image classification and object detection. In this paper, w…
Interpretable Sparsification of Brain Graphs: Better Practices and Effective Designs for Graph Neural Networks
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Georgia State University（佐治亚州立大学）；University of North Carolina, Charlotte（北卡罗来纳大学夏洛特分校）；Dartmouth College（达特茅斯学院）
Distributed scheduling algorithms for throughput or utility maximization in dense wireless multi-hop networks can have overwhelmingly high overhead, causing increased congestion, energy consumption, radio footprint, and security vulnerability. For wireless networks with dense connectivity, we propose a distributed scheme for link sparsification with graph convolutional networks (GCNs), which can reduce the scheduling overhead while keeping most of the network capacity. In a nutshell, a trainable GCN module generates node embeddings as topology-aware and reusable parameters for a local decision…
Graph neural networks (GNNs) have achieved great success on various tasks and fields that require relational modeling. GNNs aggregate node features using the graph structure as inductive biases resulting in flexible and powerful models. However, GNNs remain hard to interpret as the interplay between node features and graph structure is only implicitly learned. In this paper, we propose a novel method called Kedge for explicitly sparsifying the underlying graph by removing unnecessary neighbors. Our key idea is based on a tractable method for sparsification using the Hard Kumaraswamy distributi…
Mitigating Over-Squashing in Graph Neural Networks by Spectrum-Preserving Sparsification
Kim Jaechul Graduate School of Artificial Intelligence, KAIST, Seoul, Republic of Korea（韩国釜山科学技术院人工智能研究生院）；School of Electrical Engineering, KAIST, Daejeon, Republic of Korea（韩国釜山科学技术院电子工程学院）；Department of Engineering, University of Cambridge, Cambridge, United Kingdom（剑桥大学工程系）；Incubation Institute, Fudan University, Shanghai, China（复旦大学孵化院）；Shanghai Academy of Artificial Intelligence for Science, Shanghai, China（上海人工智能科学研究院）；School of Information and Communication Technology, Griffith University, Gold Coast, Australia（格里菲斯大学信息与通信技术学院）
The eigendeomposition of nearest-neighbor (NN) graph Laplacian matrices is the main computational bottleneck in spectral clustering. In this work, we introduce a highly-scalable, spectrum-preserving graph sparsification algorithm that enables to build ultra-sparse NN (u-NN) graphs with guaranteed preservation of the original graph spectrums, such as the first few eigenvectors of the original graph Laplacian. Our approach can immediately lead to scalable spectral clustering of large data networks without sacrificing solution quality. The proposed method starts from constructing low-stretch span…
Improved large-scale graph learning through ridge spectral sparsification
SequeL team, INRIA Lille - Nord Europe, France（SequeL团队，INRIA里尔-北欧，法国）；Facebook AI Research, Paris, France（Facebook人工智能研究，巴黎，法国）；MIT, USA（麻省理工学院，美国）；New Jersey Institute of Technology, USA（新泽西理工学院，美国）
Graph Neural Networks (GNNs) have attracted increasing attention due to its successful applications on various graph-structure data. However, recent studies have shown that adversarial attacks are threatening the functionality of GNNs. Although numerous works have been proposed to defend adversarial attacks from various perspectives, most of them can be robust against the attacks only on specific scenarios. To address this shortage of robust generalization, we propose to defend the adversarial attacks on GNN through applying the Spatio-Temporal sparsification (called ST-Sparse) on the GNN hidd…
EDiS: Edge Disjoint Subgraph Sparsification Framework for Graph Neural Networks
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Pacific Northwest National Laboratory（太平洋西北国家实验室）；Adobe Research（Adobe研究院）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Cisco（思科）；Purdue University（普渡大学）
Improving the interpretability of GNN predictions through conformal-based graph sparsification
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；McGill University（麦吉尔大学）；Saarland University（萨尔大学）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）
École Polytechnique（巴黎综合理工学院）；Institute Polytechnique de Paris（巴黎理工学院）
Message-Passing Neural Networks (MPNNs), the most prominent Graph Neural Network (GNN) framework, celebrate much success in the analysis of graph-structured data. Concurrently, the sparsification of Neural Network models attracts a great amount of academic and industrial interest. In this paper we conduct a structured, empirical study of the effect of sparsification on the trainable part of MPNNs known as the Update step. To this end, we design a series of models to successively sparsify the linear transform in the Update step. Specifically, we propose the ExpanderGNN model with a tuneable spa…
Machine Learning-based Two-Stage Graph Sparsification for the Travelling Salesman Problem
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；School of Engineering and Computer Science（工程与计算机科学学院）；Victoria University of Wellington（惠灵顿维多利亚大学）
Communication-Efficient Federated Knowledge Graph Embedding with Entity-Wise Top-K Sparsification
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Joint NTU-Webank Research Institute on Fintech, Nanyang Technological University（南洋理工大学-微众银行金融科技联合研究院）；Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly, Nanyang Technological University（南洋理工大学-英属哥伦比亚大学老年人积极生活联合卓越研究中心）
Knowledge Graph Sparsification for GNN-based Rare Disease Diagnosis
Department of Computer Science, Technical University of Munich（技术大学慕尼黑计算机科学系）；Munich Center for Machine Learning (MCML), Munich, Germany（慕尼黑机器学习中心（MCML））；Johns Hopkins University, Baltimore, Maryland, USA（约翰霍普金斯大学）
University of Catania（卡塔尼亚大学）；Nokia Bell Labs（诺基亚贝尔实验室）；Eaton Corp.（伊顿公司）；University College Dublin（都柏林大学学院）
We propose a multi-stage learning approach for pruning the search space of maximum clique enumeration, a fundamental computationally difficult problem arising in various network analysis tasks. In each stage, our approach learns the characteristics of vertices in terms of various neighborhood features and leverage them to prune the set of vertices that are likely not contained in any maximum clique. Furthermore, we demonstrate that our approach is domain independent -- the same small set of features works well on graph instances from different domain. Compared to the state-of-the-art heuristic…
