Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree $O(\text{polylog}(1/ε))$ which is $ε$-close, and similarly for any rational function there exists a ReLU network of size $O(\text{polylog}(1/ε))$ which is $ε$-close. By contrast, polynomials need degree $Ω(\text{poly}(1/ε))$ to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend exponentially on the number of layers, which is shown to be tight; in other words, a compositional representation can be beneficial even for rational functions.
University of Oxford（牛津大学）；Cornell University（康奈尔大学）
We consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds in terms of network complexity and prove that rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth. The flexibility and smoothness of rational activation functions make them an attractive alternative to ReLU, as we demonstrate with numerical experiments.
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）；Graduate Training Center of Neuroscience（神经科学研究生培训中心）
Bounded rational decision-makers transform sensory input into motor output under limited computational resources. Mathematically, such decision-makers can be modeled as information-theoretic channels with limited transmission rate. Here, we apply this formalism for the first time to multilayer feedforward neural networks. We derive synaptic weight update rules for two scenarios, where either each neuron is considered as a bounded rational decision-maker or the network as a whole. In the update rules, bounded rationality translates into information-theoretically motivated types of regularizatio…
$\mathcal{C}^1$-approximation with rational functions and rational neural networks
IDea_Lab - The Interdisciplinary Digital Lab at the University of Graz（interdisciplinary digital lab at the university of graz）；NAWI Graz；BioTechMed Graz
University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）
Intrinsic interpretability of graph neural networks (GNNs) is to find a small subset of the input graph's features -- rationale -- which guides the model prediction. Unfortunately, the leading rationalization models often rely on data biases, especially shortcut features, to compose rationales and make predictions without probing the critical and causal patterns. Moreover, such data biases easily change outside the training distribution. As a result, these models suffer from a huge drop in interpretability and predictive performance on out-of-distribution data. In this work, we propose a new s…
Rationale is defined as a subset of input features that best explains or supports the prediction by machine learning models. Rationale identification has improved the generalizability and interpretability of neural networks on vision and language data. In graph applications such as molecule and polymer property prediction, identifying representative subgraph structures named as graph rationales plays an essential role in the performance of graph neural networks. Existing graph pooling and/or distribution intervention methods suffer from lack of examples to learn to identify optimal graph ratio…
Institute for Neural Information Processing（神经信息处理研究所）；Faculty of Engineering, Computer Science, and Psychology（工程、计算机科学与心理学学院）；Ulm University（乌尔姆大学）
Bounded rationality investigates utility-optimizing decision-makers with limited information-processing power. In particular, information theoretic bounded rationality models formalize resource constraints abstractly in terms of relative Shannon information, namely the Kullback-Leibler Divergence between the agents' prior and posterior policy. Between prior and posterior lies an anytime deliberation process that can be instantiated by sample-based evaluations of the utility function through Markov Chain Monte Carlo (MCMC) optimization. The most simple model assumes a fixed prior and can relate…
Recent explainability related studies have shown that state-of-the-art DNNs do not always adopt correct evidences to make decisions. It not only hampers their generalization but also makes them less likely to be trusted by end-users. In pursuit of developing more credible DNNs, in this paper we propose CREX, which encourages DNN models to focus more on evidences that actually matter for the task at hand, and to avoid overfitting to data-dependent bias and artifacts. Specifically, CREX regularizes the training process of DNNs with rationales, i.e., a subset of features highlighted by domain exp…
Virginia Tech（弗吉尼亚理工学院暨州立大学）；University at Albany（奥尔巴尼大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
For node level graph encoding, a recent important state-of-art method is the graph convolutional networks (GCN), which nicely integrate local vertex features and graph topology in the spectral domain. However, current studies suffer from several drawbacks: (1) graph CNNs relies on Chebyshev polynomial approximation which results in oscillatory approximation at jump discontinuities; (2) Increasing the order of Chebyshev polynomial can reduce the oscillations issue, but also incurs unaffordable computational cost; (3) Chebyshev polynomials require degree $Ω$(poly(1/$ε$)) to approximate a jump si…
EECS Queen Mary University of London（伦敦玛丽女王大学电子工程与计算机科学学院）；Imperial Business School Imperial College London（帝国理工学院帝国理工商学院）
We propose a neural network approach to price EU call options that significantly outperforms some existing pricing models and comes with guarantees that its predictions are economically reasonable. To achieve this, we introduce a class of gated neural networks that automatically learn to divide-and-conquer the problem space for robust and accurate pricing. We then derive instantiations of these networks that are 'rational by design' in terms of naturally encoding a valid call option surface that enforces no arbitrage principles. This integration of human insight within data-driven learning pro…
Learn from A Rationalist: Distilling Intermediate Interpretable Rationales
Department of Computing Science, University of Alberta, Edmonton, Canada（阿尔伯塔大学计算机科学系，加拿大埃德蒙顿）；Alberta Machine Intelligence Institute, Edmonton, Canada（阿尔伯塔机器智能研究所，加拿大埃德蒙顿）
ERGNN: Spectral Graph Neural Network With Explicitly-Optimized Rational Graph Filters
Dept. of Machine Learning（机器学习系）；MBZUAI（穆罕默德·本·拉希德人工智能技术研究所）；Institute of Automation（自动化研究所）；Chinese Academy of Sciences（中国科学院）；Sun Yat-sen University（中山大学）
Rational Neural Networks have Expressivity Advantages
Center for Applied Mathematics, Cornell University, United States（应用数学中心，康奈尔大学，美国）；Department of Mathematics, Cornell University, United States（数学系，康奈尔大学，美国）
Fractional Laplace Neural Operators: Exact Architectures, an Expressivity Frontier at Criticality, and Certified Stability for Memory-Driven Network Dynamics
Key Laboratory of Mathematics Mechanization, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院数学机械化重点实验室）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（北京雁栖湖应用数学研究院）；Yau Mathematical Sciences Center and Department of Mathematics, Tsinghua University（清华大学丘成桐数学科学中心与数学系）
In this paper, we investigate the forward problems on the data-driven rational solitons for the (2+1)-dimensional KP-I equation and spin-nonlinear Schrödinger (spin-NLS) equation via the deep neural networks leaning. Moreover, the inverse problems of the (2+1)-dimensional KP-I equation and spin-NLS equation are studied via deep learning. The main idea of the data-driven forward and inverse problems is to use the deep neural networks with the activation function to approximate the solutions of the considered (2+1)-dimensional nonlinear wave equations by optimizing the chosen loss functions rela…
Technische Universität Berlin（柏林工业大学）；Universitá della Svizzera italiana（瑞士意大利语区大学）；Nanyang Technological University（南洋理工大学）；Imperial College London（帝国理工学院）；Intel Perceptual Computing（英特尔感知计算部门）
The rise of graph-structured data such as social networks, regulatory networks, citation graphs, and functional brain networks, in combination with resounding success of deep learning in various applications, has brought the interest in generalizing deep learning models to non-Euclidean domains. In this paper, we introduce a new spectral domain convolutional architecture for deep learning on graphs. The core ingredient of our model is a new class of parametric rational complex functions (Cayley polynomials) allowing to efficiently compute spectral filters on graphs that specialize on frequency…
We establish, for the first time, connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks corresp…
Classification and detection of multiple UAVs using rational Gaussian wavelet neural networks
Department of Numerical Analysis（数值分析系）；Eötvös Loránd University（欧多维奇·劳尔大学）；System and Control Laboratory（系统与控制实验室）；HUN-REN Institute for Computer Science and Control（HUN-REN计算机科学与控制研究所）；UniDistanceSuisse；Siemens Mobility Kft.（西门子移动有限公司）；HUN-REN Centre for Energy Research（HUN-REN能源研究中心）
Human-like Category Learning by Injecting Ecological Priors from Large Language Models into Neural Networks
Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）；Helmholtz Computational Health Center（亥姆霍兹计算健康中心）；Institute for Human-Centered AI（以人为本人工智能研究所）
We prove a negative result for the approximation of functions defined on compact subsets of $\mathbb{R}^d$ (where $d \geq 2$) using feedforward neural networks with one hidden layer and arbitrary continuous activation function. In a nutshell, this result claims the existence of target functions that are as difficult to approximate using these neural networks as one may want. We also demonstrate an analogous result (for general $d \in \mathbb{N}$) for neural networks with an \emph{arbitrary} number of hidden layers, for activation functions that are either rational functions or continuous splin…
Institute of Economics and L’EMbEDS, Sant’Anna School of Advanced Studies（经济学研究所和L’EMbEDS，圣安娜高级研究院）；Dept. of Statistics and Huck Institutes of the Life Sciences, Penn State University（统计系和生命科学学院，宾夕法尼亚州立大学）
Functions are rich in meaning and can be interpreted in a variety of ways. Neural networks were proven to be capable of approximating a large class of functions[1]. In this paper, we propose a new class of neural networks called "Neural Network Processing Neural Networks" (NNPNNs), which inputs neural networks and numerical values, instead of just numerical values. Thus enabling neural networks to represent and process rich structures.
Carnegie Mellon University（卡内基梅隆大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）
Neural models for NLP typically use large numbers of parameters to reach state-of-the-art performance, which can lead to excessive memory usage and increased runtime. We present a structure learning method for learning sparse, parameter-efficient NLP models. Our method applies group lasso to rational RNNs (Peng et al., 2018), a family of models that is closely connected to weighted finite-state automata (WFSAs). We take advantage of rational RNNs' natural grouping of the weights, so the group lasso penalty directly removes WFSA states, substantially reducing the number of parameters in the mod…
Discovering hidden partial differential equations (PDEs) and operators from data is an important topic at the frontier between machine learning and numerical analysis. This doctoral thesis introduces theoretical results and deep learning algorithms to learn Green's functions associated with linear partial differential equations and rigorously justify PDE learning techniques. A theoretically rigorous algorithm is derived to obtain a learning rate, which characterizes the amount of training data needed to approximately learn Green's functions associated with elliptic PDEs. The construction conne…
Adaptive Rational Activations to Boost Deep Reinforcement Learning
TU Darmstadt（达姆施塔特工业大学）；German Center for Artificial Intelligence（德国人工智能研究中心）；Hessian Center for Artifical Intelligence（黑森人工智能中心）；Centre for Cognitive Science（认知科学中心）
Numerous physical systems are described by ordinary or partial differential equations whose solutions are given by holomorphic or meromorphic functions in the complex domain. In many cases, only the magnitude of these functions are observed on various points on the purely imaginary jw-axis since coherent measurement of their phases is often expensive. However, it is desirable to retrieve the lost phases from the magnitudes when possible. To this end, we propose a physics-infused deep neural network based on the Blaschke products for phase retrieval. Inspired by the Helson and Sarason Theorem, …
The aim of this project is to develop a code to discover the optimal sigma value that maximum the F1 score and the optimal sigma value that maximizes the accuracy and to find out if they are the same. Four algorithms which can be used to solve this problem are: Genetic Regression Neural Networks (GRNNs), Radial Based Function (RBF) Neural Networks (RBFNNs), Support Vector Machines (SVMs) and Feedforward Neural Network (FFNNs).
Recently, convergence as well as convergence rate analyses of deep learning optimizers for nonconvex optimization have been widely studied. Meanwhile, numerical evaluations for the optimizers have precisely clarified the relationship between batch size and the number of steps needed for training deep neural networks. The main contribution of this paper is to show theoretically that the number of steps needed for nonconvex optimization of each of the optimizers can be expressed as a rational function of batch size. Having these rational functions leads to two particularly important facts, which…
The task of Visual Commonsense Reasoning is extremely challenging in the sense that the model has to not only be able to answer a question given an image, but also be able to learn to reason. The baselines introduced in this task are quite limiting because two networks are trained for predicting answers and rationales separately. Question and image is used as input to train answer prediction network while question, image and correct answer are used as input in the rationale prediction network. As rationale is conditioned on the correct answer, it is based on the assumption that we can solve Vi…
The Pennsylvania State University（宾夕法尼亚州立大学）；Florida International University（佛罗里达国际大学）
Uncovering rationales behind predictions of graph neural networks (GNNs) has received increasing attention over recent years. Instance-level GNN explanation aims to discover critical input elements, like nodes or edges, that the target GNN relies upon for making predictions. Though various algorithms are proposed, most of them formalize this task by searching the minimal subgraph which can preserve original predictions. However, an inductive bias is deep-rooted in this framework: several subgraphs can result in the same or similar outputs as the original graphs. Consequently, they have the dan…
Augusta University（奥古斯塔大学）；Southwest Jiaotong University（西南交通大学）
In this paper, we propose a concept of approximate bisimulation relation for feedforward neural networks. In the framework of approximate bisimulation relation, a novel neural network merging method is developed to compute the approximate bisimulation error between two neural networks based on reachability analysis of neural networks. The developed method is able to quantitatively measure the distance between the outputs of two neural networks with the same inputs. Then, we apply the approximate bisimulation relation results to perform neural networks model reduction and compute the compressio…
