We propose a Graph Neural Network with greater expressive power than commonly used GNNs - not constrained to only differentiate between graphs that Weisfeiler-Lehman test recognizes to be non-isomorphic. We use a graph attention network with expanding attention window that aggregates information from nodes exponentially far away. We also use partially random initial embeddings, allowing differentiation between nodes that would otherwise look the same. This could cause problem with a traditional dropout mechanism, therefore we use a "head dropout", randomly ignoring some attention heads rather than some dimensions of the embedding.
Towards Bridging Generalization and Expressivity of Graph Neural Networks
School of Computing, Australian National University（澳大利亚国立大学计算机学院）；Data61, CSIRO（联邦科学与工业研究组织Data61研究院）；Department of Computer Science, University of Antwerp（安特卫普大学计算机学院）；CSE & GSAI, POSTECH（浦项科技大学计算机科学与工程系及人工智能研究生院）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Cornell University（康奈尔大学）
We demonstrate the applicability of model-agnostic algorithms for meta-learning, specifically Reptile, to GNN models in molecular regression tasks. Using meta-learning we are able to learn new chemical prediction tasks with only a few model updates, as compared to using randomly initialized GNNs which require learning each regression task from scratch. We experimentally show that GNN layer expressivity is correlated to improved meta-learning. Additionally, we also experiment with GNN emsembles which yield best performance and rapid convergence for k-shot learning.
Generalization, Expressivity, and Universality of Graph Neural Networks on Attributed Graphs
Technion – IIT Faculty of Mathematics（技术学院-理工学院数学系）；MIT Dept of EECS and CSAIL（麻省理工学院电子工程与计算机科学系和计算机科学与人工智能实验室）；TUM School of CIT MCML MDSI（技术大学计算机信息科技学院MCML MDSI）
On the Rademacher Complexity of Graph Neural Networks: Unifying Expressivity and Geometry
University of Fribourg（弗里堡大学）；Johns Hopkins University（约翰·霍普金斯大学）；Université Paris-Saclay（巴黎萨克雷大学）；University of Ferrara（费拉拉大学）；Technical University of Munich（慕尼黑技术大学）；University of Siena（锡耶纳大学）
Expressivity plays a fundamental role in evaluating deep neural networks, and it is closely related to understanding the limit of performance improvement. In this paper, we propose a three-pipeline training framework based on critical expressivity, including global model contraction, weight evolution, and link's weight rewiring. Specifically, we propose a pyramidal-like skeleton to overcome the saddle points that affect information transfer. Then we analyze the reason for the modularity (clustering) phenomenon in network topology and use it to rewire potential erroneous weighted links. We cond…
Graph Representational Learning: When Does More Expressivity Hurt Generalization?
Aleph Alpha Research（Aleph Alpha 研究院）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；TU Wien（维也纳工业大学）；DLR-German Aerospace Center（德国航空航天中心）；University of Tromsø（特罗姆瑟大学）；CNRS（法国国家科学研究中心）；Institut Polytechnique de Paris（巴黎理工学院）
A Unifying Relational Perspective on Expressive Lottery Tickets
Faculty of Computer Science, University of Vienna（维也纳大学计算机科学学院）；Doctoral School Computer Science, University of Vienna（维也纳大学计算机科学博士学院）；CeMM Research Center for Molecular Medicine of the Austrian Academy of Sciences（奥地利科学院CeMM分子医学研究中心）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦大学玛丽女王学院电子工程与计算机科学学院）；Research Network Data Science, University of Vienna（维也纳大学数据科学研究网络）
Identifying similar network structures is key to capture graph isomorphisms and learn representations that exploit structural information encoded in graph data. This work shows that ego-networks can produce a structural encoding scheme for arbitrary graphs with greater expressivity than the Weisfeiler-Lehman (1-WL) test. We introduce IGEL, a preprocessing step to produce features that augment node representations by encoding ego-networks into sparse vectors that enrich Message Passing (MP) Graph Neural Networks (GNNs) beyond 1-WL expressivity. We describe formally the relation between IGEL and…
Expressivity and Generalization: Fragment-Biases for Molecular GNNs
School of Computation, Information & Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；Munich Data Science Institute（慕尼黑数据科学研究所）；Helmholtz Center for Computational Health（亥姆霍兹计算健康中心）
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Ecole Polytechnique de Montréal（蒙特利尔高等理工学院）
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the cost of reduced expressivity due to the limited variety of orthogonal transformations. We propose a novel connectivity structure based on the Schur decomposition and a splitting of the Schur form into normal and non-normal parts. This allows to parametrize matri…
Spectral Graph Neural Networks are Incomplete on Graphs with a Simple Spectrum
Snir Hordan Faculty of Mathematics Department of Applied Mathematics Technion - Israel Institute of Technology（技术学院-以色列理工学院数学系应用数学系）；Maya Bechler-Speicher Meta（Meta公司）；Gur Lifshitz Blavatnik School of Computer Science Tel-Aviv University（特拉维夫大学Blavatnik计算机科学学院）；Nadav Dym Faculty of Mathematics Faculty of Computer Science Technion - Israel Institute of Technology（技术学院-以色列理工学院数学系计算机科学系）
Logical Expressivity and Explanations for Monotonic GNNs with Scoring Functions
Department of Compute Science, University of Oxford（牛津大学计算机科学系）；Department of Computer Science, Royal Holloway, University of London（伦敦大学皇家霍洛威学院计算机科学系）
Institute of Mathematics, Technical University of Berlin（柏林工业大学数学研究所）
In this review paper, we give a comprehensive overview of the large variety of approximation results for neural networks. Approximation rates for classical function spaces as well as benefits of deep neural networks over shallow ones for specifically structured function classes are discussed. While the mainbody of existing results is for general feedforward architectures, we also depict approximation results for convolutional, residual and recurrent neural networks.
Google Brain（谷歌大脑）；Cornell University（康奈尔大学）；Stanford University（斯坦福大学）
We survey results on neural network expressivity described in "On the Expressive Power of Deep Neural Networks". The paper motivates and develops three natural measures of expressiveness, which all display an exponential dependence on the depth of the network. In fact, all of these measures are related to a fourth quantity, trajectory length. This quantity grows exponentially in the depth of the network, and is responsible for the depth sensitivity observed. These results translate to consequences for networks during and after training. They suggest that parameters earlier in a network have gr…
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of Artificial Intelligence（初探人工智能研究所）
One fundamental problem in deep learning is understanding the outstanding performance of deep Neural Networks (NNs) in practice. One explanation for the superiority of NNs is that they can realize a large class of complicated functions, i.e., they have powerful expressivity. The expressivity of a ReLU NN can be quantified by the maximal number of linear regions it can separate its input space into. In this paper, we provide several mathematical results needed for studying the linear regions of CNNs, and use them to derive the maximal and average numbers of linear regions for one-layer ReLU CNN…
Shanghai Jiao Tong University（上海交通大学）；NVIDIA Corporation（英伟达公司）；University of California（加利福尼亚大学）
Expressivity is one of the most significant issues in assessing neural networks. In this paper, we provide a quantitative analysis of the expressivity for the deep neural network (DNN) from its dynamic model, where the Hilbert space is employed to analyze the convergence and criticality. We study the feature mapping of several widely used activation functions obtained by Hermite polynomials, and find sharp declines or even saddle points in the feature space, which stagnate the information transfer in DNNs. We then present a new activation function design based on the Hermite polynomials for be…
Institute for Theoretical Computer Science（理论计算机科学研究所）；Leibniz University Hanover（汉诺威莱布尼茨大学）；School of Computer Science（计算机科学学院）；University of Sheffield（谢菲尔德大学）
We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results reveal an order-to-chaos expressivity phase transition, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth but not width. We prove this generic class of deep random functions cannot be efficiently computed by any shallow network, going beyond prior work restricted to the analysis of single functions. Moreover, we formalize and quantitativ…
We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.
Fractional Laplace Neural Operators: Exact Architectures, an Expressivity Frontier at Criticality, and Certified Stability for Memory-Driven Network Dynamics
