Efficient numerical solvers for sparse linear systems are crucial in science and engineering. One of the fastest methods for solving large-scale sparse linear systems is algebraic multigrid (AMG). The main challenge in the construction of AMG algorithms is the selection of the prolongation operator -- a problem-dependent sparse matrix which governs the multiscale hierarchy of the solver and is critical to its efficiency. Over many years, numerous methods have been developed for this task, and yet there is no known single right answer except in very special cases. Here we propose a framework for learning AMG prolongation operators for linear systems with sparse symmetric positive (semi-) definite matrices. We train a single graph neural network to learn a mapping from an entire class of such matrices to prolongation operators, using an efficient unsupervised loss function. Experiments on a broad class of problems demonstrate improved convergence rates compared to classical AMG, demonstrating the potential utility of neural networks for developing sparse system solvers.
RAPNet: Accelerating Algebraic Multigrid with Learned Sparse Corrections
Institute for Interdisciplinary Computational Sciences, Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Be'er Sheva, Israel（交叉学科计算科学研究所，计算机与信息科学学院，内盖夫本·古里安大学，以色列贝尔谢瓦）；Department of Mathematics and Computer Science, Emory University, Atlanta, GA, USA（数学与计算机科学系，埃默里大学，美国亚特兰大，GA）
Acceleration of an algebraic multigrid pressure solver using graph neural networks
Faculty of Mechanical Engineering, Delft University of Technology, The Netherlands（荷兰代尔夫特理工大学机械工程学院）；Maritime Research Institute Netherlands, The Netherlands（荷兰海事研究院）；Department of Aeronautics, Imperial College London, United Kingdom（英国伦敦帝国理工学院航空系）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Memorial University of Newfoundland（纽芬兰纪念大学）
Domain decomposition methods (DDMs) are popular solvers for discretized systems of partial differential equations (PDEs), with one-level and multilevel variants. These solvers rely on several algorithmic and mathematical parameters, prescribing overlap, subdomain boundary conditions, and other properties of the DDM. While some work has been done on optimizing these parameters, it has mostly focused on the one-level setting or special cases such as structured-grid discretizations with regular subdomain construction. In this paper, we propose multigrid graph neural networks (MG-GNN), a novel GNN…
NeuraLSP: A Neural Spectral Preconditioner for Accelerating PDE Solvers
Department of Applied Mathematics and Statistics, Stony Brook University（应用数学与统计学系，史泰尼斯布鲁克大学）；Department of Computer Science, Stony Brook University（计算机科学系，史泰尼斯布鲁克大学）
Agglomeration-based strategies are important both within adaptive refinement algorithms and to construct scalable multilevel algebraic solvers. In order to automatically perform agglomeration of polygonal grids, we propose the use of Machine Learning (ML) strategies, that can naturally exploit geometrical information about the mesh in order to preserve the grid quality, enhancing performance of numerical methods and reducing the overall computational cost. In particular, we employ the k-means clustering algorithm and Graph Neural Networks (GNNs) to partition the connectivity graph of a computa…
The support vector machine is a flexible optimization-based technique widely used for classification problems. In practice, its training part becomes computationally expensive on large-scale data sets because of such reasons as the complexity and number of iterations in parameter fitting methods, underlying optimization solvers, and nonlinearity of kernels. We introduce a fast multilevel framework for solving support vector machine models that is inspired by the algebraic multigrid. Significant improvement in the running has been achieved without any loss in the quality. The proposed technique…
Iowa State University（爱荷华州立大学）；RocketML Inc.（RocketML公司）；New York University（纽约大学）
We consider the distributed training of large-scale neural networks that serve as PDE solvers producing full field outputs. We specifically consider neural solvers for the generalized 3D Poisson equation over megavoxel domains. A scalable framework is presented that integrates two distinct advances. First, we accelerate training a large model via a method analogous to the multigrid technique used in numerical linear algebra. Here, the network is trained using a hierarchy of increasing resolution inputs in sequence, analogous to the 'V', 'W', 'F', and 'Half-V' cycles used in multigrid approache…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Memorial University of Newfoundland（纽芬兰纪念大学）
Large sparse linear systems of equations are ubiquitous in science and engineering, such as those arising from discretizations of partial differential equations. Algebraic multigrid (AMG) methods are one of the most common methods of solving such linear systems, with an extensive body of underlying mathematical theory. A system of linear equations defines a graph on the set of unknowns and each level of a multigrid solver requires the selection of an appropriate coarse graph along with restriction and interpolation operators that map to and from the coarse representation. The efficiency of the…
Graph Convolutional Networks (GCNs) have shown to be effective in handling unordered data like point clouds and meshes. In this work we propose novel approaches for graph convolution, pooling and unpooling, inspired from finite differences and algebraic multigrid frameworks. We form a parameterized convolution kernel based on discretized differential operators, leveraging the graph mass, gradient and Laplacian. This way, the parameterization does not depend on the graph structure, only on the meaning of the network convolutions as differential operators. To allow hierarchical representations o…
Massachusetts Institute of Technology Lincoln Laboratory Supercomputing Center（麻省理工学院林肯实验室超级计算中心）
A Multigrid Full Approximation Storage algorithm for solving Deep Residual Networks is developed to enable neural network parallelized layer-wise training and concurrent computational kernel execution on GPUs. This work demonstrates a 10.2x speedup over traditional layer-wise model parallelism techniques using the same number of compute units.
We present a multigrid approach that combats the quadratic growth of the number of parameters with respect to the number of channels in standard convolutional neural networks (CNNs). It has been shown that there is a redundancy in standard CNNs, as networks with much sparser convolution operators can yield similar performance to full networks. The sparsity patterns that lead to such behavior, however, are typically random, hampering hardware efficiency. In this work, we present a multigrid-in-channels approach for building CNN architectures that achieves full coupling of the channels, and whos…
We propose a multigrid extension of convolutional neural networks (CNNs). Rather than manipulating representations living on a single spatial grid, our network layers operate across scale space, on a pyramid of grids. They consume multigrid inputs and produce multigrid outputs; convolutional filters themselves have both within-scale and cross-scale extent. This aspect is distinct from simple multiscale designs, which only process the input at different scales. Viewed in terms of information flow, a multigrid network passes messages across a spatial pyramid. As a consequence, receptive field si…
A level-wise training scheme for learning neural multigrid smoothers with application to integral equations
Hetao Institute of Mathematics and Interdisciplinary Sciences（河套数学与交叉科学研究所）；Hong Kong University of Science and Technology（香港科技大学）；Lingnan University（岭南大学）；University of Waterloo（多伦多大学）
Emory University（埃默里大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Multigrid methods are one of the most efficient techniques for solving linear systems arising from Partial Differential Equations (PDEs) and graph Laplacians from machine learning applications. One of the key components of multigrid is smoothing, which aims at reducing high-frequency errors on each grid level. However, finding optimal smoothing algorithms is problem-dependent and can impose challenges for many problems. In this paper, we propose an efficient adaptive framework for learning optimized smoothers from operator stencils in the form of convolutional neural networks (CNNs). The CNNs …
Algebraic or geometric multigrid methods are commonly used in numerical solvers as they are a multi-resolution method able to handle problems with multiple scales. In this work, we propose a modification to the commonly-used U-Net neural network architecture that is inspired by the principles of multigrid methods, referred to here as U-Net-MG. We then demonstrate that this proposed U-Net-MG architecture can successfully reduce the test prediction errors relative to the conventional U-Net architecture when modeling a set of fluid dynamic problems. In total, we demonstrate an improvement in the …
University of Chicago（芝加哥大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）
We introduce a novel approach to endowing neural networks with emergent, long-term, large-scale memory. Distinct from strategies that connect neural networks to external memory banks via intricately crafted controllers and hand-designed attentional mechanisms, our memory is internal, distributed, co-located alongside computation, and implicitly addressed, while being drastically simpler than prior efforts. Architecting networks with multigrid structure and connectivity, while distributing memory cells alongside computation throughout this topology, we observe the emergence of coherent memory s…
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；Emory University（埃默里大学）
We present a multigrid-in-channels (MGIC) approach that tackles the quadratic growth of the number of parameters with respect to the number of channels in standard convolutional neural networks (CNNs). Thereby our approach addresses the redundancy in CNNs that is also exposed by the recent success of lightweight CNNs. Lightweight CNNs can achieve comparable accuracy to standard CNNs with fewer parameters; however, the number of weights still scales quadratically with the CNN's width. Our MGIC architectures replace each CNN block with an MGIC counterpart that utilizes a hierarchy of nested grou…
In this paper, we investigate the combination of multigrid methods and neural networks, starting from a Finite Element discretization of an elliptic PDE. Multigrid methods use interpolation operators to transfer information between different levels of approximation. These operators are crucial for fast convergence of multigrid, but they are generally unknown. We propose Deep Neural Network models for learning interpolation operators and we build a multilevel hierarchy based on the output of the network. We investigate the accuracy of the interpolation operator predicted by the Neural Network, …
UGrid: An Efficient-And-Rigorous Neural Multigrid Solver for Linear PDEs
Stony Brook University (SUNY)（纽约州立大学石溪分校）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Neural Multigrid Memory For Computational Fluid Dynamics
Ecole Polytechnique（巴黎综合理工学院）；University of Engineering and Technology, Vietnam National University, Hanoi（河内越南国立大学工程技术大学）；Microsoft（微软）；OpenAI；Google（谷歌）；Littelfuse（力特公司）；Halıcıoğlu Data Science Institute, University of California, San Diego（加州大学圣地亚哥分校哈利哲奥卢数据科学研究所）
University of Wuppertal（伍珀塔尔大学）；EPFL Lausanne（洛桑联邦理工学院）
Current state-of-the-art deep neural networks for image classification are made up of 10 - 100 million learnable weights and are therefore inherently prone to overfitting. The complexity of the weight count can be seen as a function of the number of channels, the spatial extent of the input and the number of layers of the network. Due to the use of convolutional layers the scaling of weight complexity is usually linear with regards to the resolution dimensions, but remains quadratic with respect to the number of channels. Active research in recent years in terms of using multigrid inspired ide…
Weizmann Institute of Science（魏茨曼科学研究所）；Technion, Israel Institute of Technology（以色列理工学院）
Constructing fast numerical solvers for partial differential equations (PDEs) is crucial for many scientific disciplines. A leading technique for solving large-scale PDEs is using multigrid methods. At the core of a multigrid solver is the prolongation matrix, which relates between different scales of the problem. This matrix is strongly problem-dependent, and its optimal construction is critical to the efficiency of the solver. In practice, however, devising multigrid algorithms for new problems often poses formidable challenges. In this paper we propose a framework for learning multigrid sol…
