Graph neural networks (GNNs) are one of the most popular approaches to using deep learning on graph-structured data, and they have shown state-of-the-art performances on a variety of tasks. However, according to a recent study, a careful choice of pooling functions, which are used for the aggregation and readout operations in GNNs, is crucial for enabling GNNs to extrapolate. Without proper choices of pooling functions, which varies across tasks, GNNs completely fail to generalize to out-of-distribution data, while the number of possible choices grows exponentially with the number of layers. In this paper, we present GNP, a $L^p$ norm-like pooling function that is trainable end-to-end for any given task. Notably, GNP generalizes most of the widely-used pooling functions. We verify experimentally that simply using GNP for every aggregation and readout operation enables GNNs to extrapolate well on many node-level, graph-level, and set-related tasks; and GNP sometimes performs even better than the best-performing choices among existing pooling functions.
Physical Pooling Functions in Graph Neural Networks for Molecular Property Prediction
RWTH Aachen University（亚琛工业大学）；Delft University of Technology（代尔夫特理工大学）；Forschungszentrum Jülich GmbH（于利希研究中心）；JARA Center for Simulation and Data Science (CSD)（JARA模拟与数据科学中心）
Geometry-Aware Edge Pooling for Graph Neural Networks
Helmholtz Munich（海德堡-穆恩大学）；Technical University of Munich（慕尼黑技术大学）；Université de Montréal（蒙特利尔大学）；Mila - Quebec AI Institute（魁北克人工智能研究所）；Université de Fribourg（弗里堡大学）
Graph Neural Processes for Spatio-Temporal Extrapolation
National University of Singapore（新加坡国立大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Technology Sydney（悉尼科技大学）；Zhejiang Lab（之江实验室）；JD Intelligent Cities Research & JD iCity, JD Technology（京东科技京东智能城市研究院）
MaxCutPool: differentiable feature-aware Maxcut for pooling in graph neural networks
Alma Mater Studiorum - University of Bologna（博洛尼亚大学）；UiT the Arctic University of Norway（挪威北极大学）；Fondazione Istituto Italiano di Tecnologia（意大利技术研究院）；NORCE Norwegian Research Centre AS（NORCE挪威研究中心）
fortiss GmbH（fortiss有限公司）；An-Institut Technische Universität München（慕尼黑工业大学附属研究所）
Graph Neural Network (GNN) research has concentrated on improving convolutional layers, with little attention paid to developing graph pooling layers. Yet pooling layers can enable GNNs to reason over abstracted groups of nodes instead of single nodes. To close this gap, we propose a graph pooling layer relying on the notion of edge contraction: EdgePool learns a localized and sparse hard pooling transform. We show that EdgePool outperforms alternative pooling methods, can be easily integrated into most GNN models, and improves performance on both node and graph classification.
Graph neural networks have achieved great success in learning node representations for graph tasks such as node classification and link prediction. Graph representation learning requires graph pooling to obtain graph representations from node representations. It is challenging to develop graph pooling methods due to the variable sizes and isomorphic structures of graphs. In this work, we propose to use second-order pooling as graph pooling, which naturally solves the above challenges. In addition, compared to existing graph pooling methods, second-order pooling is able to use information from …
Graph Neural Networks achieve state-of-the-art performance on a plethora of graph classification tasks, especially due to pooling operators, which aggregate learned node embeddings hierarchically into a final graph representation. However, they are not only questioned by recent work showing on par performance with random pooling, but also ignore completely higher-order connectivity patterns. To tackle this issue, we propose HoscPool, a clustering-based graph pooling operator that captures higher-order information hierarchically, leading to richer graph representations. In fact, we learn a prob…
Graph neural networks (GNN) extends deep learning to graph-structure dataset. Similar to Convolutional Neural Networks (CNN) using on image prediction, convolutional and pooling layers are the foundation to success for GNN on graph prediction tasks. In the initial PAN paper, it uses a path integral based graph neural networks for graph prediction. Specifically, it uses a convolution operation that involves every path linking the message sender and receiver with learnable weights depending on the path length, which corresponds to the maximal entropy random walk. It further generalizes such conv…
Università della Svizzera italiana（瑞士意大利语大学）；UiT the Arctic University of Norway（挪威北极大学）；NORCE, The Norwegian Research Centre（挪威NORCE研究中心）；Politecnico di Milano（米兰理工大学）
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）
Graph neural networks (GNNs) have been widely used to learn vector representation of graph-structured data and achieved better task performance than conventional methods. The foundation of GNNs is the message passing procedure, which propagates the information in a node to its neighbors. Since this procedure proceeds one step per layer, the range of the information propagation among nodes is small in the lower layers, and it expands toward the higher layers. Therefore, a GNN model has to be deep enough to capture global structural information in a graph. On the other hand, it is known that dee…
NORCE, the Norwegian Research Centre（挪威研究中心NORCE）；Università della Svizzera italiana（意大利语区大学（瑞士））；Politecnico di Milano（米兰理工大学）
Spectral clustering (SC) is a popular clustering technique to find strongly connected communities on a graph. SC can be used in Graph Neural Networks (GNNs) to implement pooling operations that aggregate nodes belonging to the same cluster. However, the eigendecomposition of the Laplacian is expensive and, since clustering results are graph-specific, pooling methods based on SC must perform a new optimization for each new sample. In this paper, we propose a graph clustering approach that addresses these limitations of SC. We formulate a continuous relaxation of the normalized minCUT problem an…
McGill University（麦吉尔大学）；Université de Montréal（蒙特利尔大学）；HEC Montréal（蒙特利尔高等商学院）；Mila（米拉人工智能研究所）
Graph Neural Networks (GNNs) are attracting growing attention due to their effectiveness and flexibility in modeling a variety of graph-structured data. Exiting GNN architectures usually adopt simple pooling operations (eg. sum, average, max) when aggregating messages from a local neighborhood for updating node representation or pooling node representations from the entire graph to compute the graph representation. Though simple and effective, these linear operations do not model high-order non-linear interactions among nodes. We propose the Tensorized Graph Neural Network (tGNN), a highly exp…
Graph neural networks (GNNs) have been used effectively in different applications involving the processing of signals on irregular structures modeled by graphs. Relying on the use of shift-invariant graph filters, GNNs extend the operation of convolution to graphs. However, the operations of pooling and sampling are still not clearly defined and the approaches proposed in the literature either modify the graph structure in a way that does not preserve its spectral properties, or require defining a policy for selecting which nodes to keep. In this work, we propose a new strategy for pooling and…
Institute of Exact and Applied Sciences (EA7484), University of New Caledonia（精确与应用科学研究所（EA7484），新喀里多尼亚大学）；Big Data Institute, College of Computer Science and Software Engineering（大数据研究所，计算机科学与软件工程学院）；Shenzhen University, China（深圳大学，中国）
Nanjing Normal University（南京师范大学）；Universität Basel（巴塞尔大学）
Graph neural networks achieve high accuracy in link prediction by jointly leveraging graph topology and node attributes. Topology, however, is represented indirectly; state-of-the-art methods based on subgraph classification label nodes with distance to the target link, so that, although topological information is present, it is tempered by pooling. This makes it challenging to leverage features like loops and motifs associated with network formation mechanisms. We propose a link prediction algorithm based on a new pooling scheme called WalkPool. WalkPool combines the expressivity of topologic…
Graph Neural Networks (GNNs), which generalize deep neural networks to graph-structured data, have drawn considerable attention and achieved state-of-the-art performance in numerous graph related tasks. However, existing GNN models mainly focus on designing graph convolution operations. The graph pooling (or downsampling) operations, that play an important role in learning hierarchical representations, are usually overlooked. In this paper, we propose a novel graph pooling operator, called Hierarchical Graph Pooling with Structure Learning (HGP-SL), which can be integrated into various graph n…
Michigan State University（密歇根州立大学）；Pennsylvania State University（宾夕法尼亚州立大学）；IBM T. J. Watson Research Center（IBM T·J·沃森研究中心）
Graph neural networks, which generalize deep neural network models to graph structured data, have attracted increasing attention in recent years. They usually learn node representations by transforming, propagating and aggregating node features and have been proven to improve the performance of many graph related tasks such as node classification and link prediction. To apply graph neural networks for the graph classification task, approaches to generate the \textit{graph representation} from node representations are demanded. A common way is to globally combine the node representations. Howev…
Aalto University（阿尔托大学）；Federal Institute of Ceará（塞阿拉联邦研究所）；University of Manchester（曼彻斯特大学）
Graph pooling is a central component of a myriad of graph neural network (GNN) architectures. As an inheritance from traditional CNNs, most approaches formulate graph pooling as a cluster assignment problem, extending the idea of local patches in regular grids to graphs. Despite the wide adherence to this design choice, no work has rigorously evaluated its influence on the success of GNNs. In this paper, we build upon representative GNNs and introduce variants that challenge the need for locality-preserving representations, either using randomization or clustering on the complement graph. Stri…
Hierarchical Pooling and Explainability in Graph Neural Networks for Tumor and Tissue-of-Origin Classification Using RNA-seq Data
Institute of Informatics（信息学院）；Universidade Federal do Rio Grande do Sul（鲁汶联邦大学）；Bioinformatics Core（生物信息学核心）；Hospital de Clínicas de Porto Alegre（波多利亚临床医院）
This work generalizes graph neural networks (GNNs) beyond those based on the Weisfeiler-Lehman (WL) algorithm, graph Laplacians, and diffusions. Our approach, denoted Relational Pooling (RP), draws from the theory of finite partial exchangeability to provide a framework with maximal representation power for graphs. RP can work with existing graph representation models and, somewhat counterintuitively, can make them even more powerful than the original WL isomorphism test. Additionally, RP allows architectures like Recurrent Neural Networks and Convolutional Neural Networks to be used in a theo…
Yale University（耶鲁大学）；Yale School of Medicine（耶鲁大学医学院）；Facebook AI Research（Facebook人工智能研究院）
Understanding how certain brain regions relate to a specific neurological disorder has been an important area of neuroimaging research. A promising approach to identify the salient regions is using Graph Neural Networks (GNNs), which can be used to analyze graph structured data, e.g. brain networks constructed by functional magnetic resonance imaging (fMRI). We propose an interpretable GNN framework with a novel salient region selection mechanism to determine neurological brain biomarkers associated with disorders. Specifically, we design novel regularized pooling layers that highlight salient…
ENADPool: The Edge-Node Attention-based Differentiable Pooling for Graph Neural Networks
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Information, Central University of Finance and Economics（中央财经大学信息学院）；Key Laboratory of Intelligent Education Technology and Application of Zhejiang Province, Zhejiang Normal University（浙江师范大学浙江省智能教育技术与应用重点实验室）；Department of Computer Science, University of York（约克大学计算机科学系）
In this paper, we propose a novel pooling layer for graph neural networks based on maximizing the mutual information between the pooled graph and the input graph. Since the maximum mutual information is difficult to compute, we employ the Shannon capacity of a graph as an inductive bias to our pooling method. More precisely, we show that the input graph to the pooling layer can be viewed as a representation of a noisy communication channel. For such a channel, sending the symbols belonging to an independent set of the graph yields a reliable and error-free transmission of information. We show …
Student performance prediction is critical to online education. It can benefit many downstream tasks on online learning platforms, such as estimating dropout rates, facilitating strategic intervention, and enabling adaptive online learning. Interactive online question pools provide students with interesting interactive questions to practice their knowledge in online education. However, little research has been done on student performance prediction in interactive online question pools. Existing work on student performance prediction targets at online learning platforms with predefined course c…
Beihang University（北京航空航天大学）；Macquarie University（麦考瑞大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Lehigh University（理海大学）
Graph representation learning has attracted increasing research attention. However, most existing studies fuse all structural features and node attributes to provide an overarching view of graphs, neglecting finer substructures' semantics, and suffering from interpretation enigmas. This paper presents a novel hierarchical subgraph-level selection and embedding based graph neural network for graph classification, namely SUGAR, to learn more discriminative subgraph representations and respond in an explanatory way. SUGAR reconstructs a sketched graph by extracting striking subgraphs as the repre…
Prior studies using graph neural networks (GNNs) for image classification have focused on graphs generated from a regular grid of pixels or similar-sized superpixels. In the latter, a single target number of superpixels is defined for an entire dataset irrespective of differences across images and their intrinsic multiscale structure. On the contrary, this study investigates image classification using graphs generated from an image-specific number of multiscale superpixels. We propose WaveMesh, a new wavelet-based superpixeling algorithm, where the number and sizes of superpixels in an image a…
Shanghai Jiao Tong University（上海交通大学）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL））
With the advent of data science, the analysis of network or graph data has become a very timely research problem. A variety of recent works have been proposed to generalize neural networks to graphs, either from a spectral graph theory or a spatial perspective. The majority of these works however focus on adapting the convolution operator to graph representation. At the same time, the pooling operator also plays an important role in distilling multiscale and hierarchical representations but it has been mostly overlooked so far. In this paper, we propose a parameter-free pooling operator, calle…
Spatiotemporal graph neural process for reconstruction, extrapolation, and classification of cardiac trajectories
Department of Radiology, Lausanne University Hospital（放射科，洛桑大学医院）；University of Lausanne（洛桑大学）；Cardiovascular Department, Lausanne University Hospital（心血管科，洛桑大学医院）；Department of Medicine, Geneva University Hospital（医学部，日内瓦大学医院）；University of Geneva（日内瓦大学）；CIBM Center for Biomedical Imaging（生物医学成像中心）
University of Pittsburgh（匹兹堡大学）；Intel Labs（英特尔实验室）
Recent years have witnessed the emergence and development of graph neural networks (GNNs), which have been shown as a powerful approach for graph representation learning in many tasks, such as node classification and graph classification. The research on the robustness of these models has also started to attract attentions in the machine learning field. However, most of the existing work in this area focus on the GNNs for node-level tasks, while little work has been done to study the robustness of the GNNs for the graph classification task. In this paper, we aim to explore the vulnerability of…
Hierarchical Representation Learning in Graph Neural Networks with Node Decimation Pooling
UiT the Arctic University of Norway（挪威北极大学）；NORCE, Norwegian Research Centre（挪威NORCE研究中心）；Università della Svizzera italiana（意大利语瑞士大学）；University of Manitoba（曼尼托巴大学）；University of Exeter（埃克塞特大学）；Politecnico di Milano（米兰理工大学）
Princeton University（普林斯顿大学）；University of Technology Sydney（悉尼科技大学）；Max Planck Institute for Mathematics in the Sciences（马克斯·普朗克数学科学研究所）；University of New South Wales（新南威尔士大学）；Zhejiang Normal University（浙江师范大学）；University of Cambridge（剑桥大学）
Graph neural networks (GNNs) extends the functionality of traditional neural networks to graph-structured data. Similar to CNNs, an optimized design of graph convolution and pooling is key to success. Borrowing ideas from physics, we propose a path integral based graph neural networks (PAN) for classification and regression tasks on graphs. Specifically, we consider a convolution operation that involves every path linking the message sender and receiver with learnable weights depending on the path length, which corresponds to the maximal entropy random walk. It generalizes the graph Laplacian …
We propose PiNet, a generalised differentiable attention-based pooling mechanism for utilising graph convolution operations for graph level classification. We demonstrate high sample efficiency and superior performance over other graph neural networks in distinguishing isomorphic graph classes, as well as competitive results with state of the art methods on standard chemo-informatics datasets.
IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Northwestern University（西北大学）
Recent advances in the integration of deep learning with automated theorem proving have centered around the representation of logical formulae as inputs to deep learning systems. In particular, there has been a growing interest in adapting structure-aware neural methods to work with the underlying graph representations of logical expressions. While more effective than character and token-level approaches, graph-based methods have often made representational trade-offs that limited their ability to capture key structural properties of their inputs. In this work we propose a novel approach for e…
Deep learning inspired by differential equations is a recent research trend and has marked the state of the art performance for many machine learning tasks. Among them, time-series modeling with neural controlled differential equations (NCDEs) is considered as a breakthrough. In many cases, NCDE-based models not only provide better accuracy than recurrent neural networks (RNNs) but also make it possible to process irregular time-series. In this work, we enhance NCDEs by redesigning their core part, i.e., generating a continuous path from a discrete time-series input. NCDEs typically use interp…
Architectural change in neural networks using fuzzy vertex pooling
Department of General Science, Birla Institute of Technology and Science Pilani, International Academic City（一般科学系，比拉理工学院和科学学院，国际学术城）；Department of Mathematics, Providence Women’s College, Calicut, India（数学系，普罗维登斯女子学院，卡利卡特，印度）；Department of Mathematics, National Institute of Technology, Calicut, India（数学系，国家理工学院，卡利卡特，印度）
We propose a novel pooling strategy that learns how to adaptively rank deep convolutional features for selecting more informative representations. To this end, we exploit discriminative analysis to project the features onto a space spanned by the number of classes in the dataset under study. This maps the notion of labels in the feature space into instances in the projected space. We employ these projected distances as a measure to rank the existing features with respect to their specific discriminant power for each individual class. We then apply multipartite ranking to score the separability…
Handong Global University（韩东国际大学）；Dept. of Information and Communication Engineering（信息与通信工程系）
Convolutional neural networks (CNNs) have achieved remarkable performance in many applications, especially in image recognition tasks. As a crucial component of CNNs, sub-sampling plays an important role for efficient training or invariance property, and max-pooling and arithmetic average-pooling are commonly used sub-sampling methods. In addition to the two pooling methods, however, there could be many other pooling types, such as geometric average, harmonic average, and so on. Since it is not easy for algorithms to find the best pooling method, usually the pooling types are assumed a priorit…
Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks (PNNs). In this work, we derive the finite-width NTK formulation for a special class of NNs-Hp, i.e., polynomial neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK. Based on our res…
Image classification is considered, and a hierarchical max-pooling model with additional local pooling is introduced. Here the additional local pooling enables the hierachical model to combine parts of the image which have a variable relative distance towards each other. Various convolutional neural network image classifiers are introduced and compared in view of their rate of convergence. The finite sample size performance of the estimates is analyzed by applying them to simulated and real data.
Pooling operations, which can be calculated at low cost and serve as a linear or nonlinear transfer function for data reduction, are found in almost every modern neural network. Countless modern approaches have already tackled replacing the common maximum value selection and mean value operations, not to mention providing a function that allows different functions to be selected through changing parameters. Additional neural networks are used to estimate the parameters of these pooling functions.Consequently, pooling layers may require supplementary parameters to increase the complexity of the…
Graph convolutional neural networks (GCNNs) are a powerful extension of deep learning techniques to graph-structured data problems. We empirically evaluate several pooling methods for GCNNs, and combinations of those graph pooling methods with three different architectures: GCN, TAGCN, and GraphSAGE. We confirm that graph pooling, especially DiffPool, improves classification accuracy on popular graph classification datasets and find that, on average, TAGCN achieves comparable or better accuracy than GCN and GraphSAGE, particularly for datasets with larger and sparser graph structures.
Shahrood University of Technology（沙赫鲁德工业大学）；Faculty of Electrical & Robotics Engineering（电气与机器人工程学院）
Nowadays, Deep Neural Networks are among the main tools used in various sciences. Convolutional Neural Network is a special type of DNN consisting of several convolution layers, each followed by an activation function and a pooling layer. The pooling layer is an important layer that executes the down-sampling on the feature maps coming from the previous layer and produces new feature maps with a condensed resolution. This layer drastically reduces the spatial dimension of input. It serves two main purposes. The first is to reduce the number of parameters or weights, thus lessening the computat…
Colorado State University（科罗拉多州立大学）；The Ohio State University（俄亥俄州立大学）；Boise State University（博伊西州立大学）；Cooperative Institute for Research in the Atmosphere（大气研究合作研究所）
