Rational neural networks

有理ANOVA网络

Rational ANOVA Networks

有理ANOVA网络通过有理逼近和ANOVA分解提升深度学习的可解释性和外推能力。

Negative results for approximation using single layer and multilayer feedforward neural networks

We prove a negative result for the approximation of functions defined on compact subsets of $\mathbb{R}^d$ (where $d \geq 2$) using feedforward neural networks with one hidden layer and arbitrary continuous activation function. In a nutshell, this result claims the existence of target functions that are as difficult to approximate using these neural networks as one may want. We also demonstrate an analogous result (for general $d \in \mathbb{N}$) for neural networks with an \emph{arbitrary} number of hidden layers, for activation functions that are either rational functions or continuous splin…

Neural networks and rational functions

Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree $O(\text{polylog}(1/ε))$ which is $ε$-close, and similarly for any rational function there exists a ReLU network of size $O(\text{polylog}(1/ε))$ which is $ε$-close. By contrast, polynomials need degree $Ω(\text{poly}(1/ε))$ to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend exponentially on the number of layers, which is shown to be tight; in …

计算异或的单层Transformer的下界

Lower bounds for one-layer transformers that compute parity

该研究证明，除非头数与后处理函数次数的乘积与输入长度线性增长，否则无法用有理函数后处理的自注意力层表示异或函数，并扩展了ReLU网络的自注意力层。

Bounded Rational Decision-Making in Feedforward Neural Networks

Bounded rational decision-makers transform sensory input into motor output under limited computational resources. Mathematically, such decision-makers can be modeled as information-theoretic channels with limited transmission rate. Here, we apply this formalism for the first time to multilayer feedforward neural networks. We derive synaptic weight update rules for two scenarios, where either each neuron is considered as a bounded rational decision-maker or the network as a whole. In the update rules, bounded rationality translates into information-theoretically motivated types of regularizatio…

从频率视角理解神经网络训练中的grokking现象的理论依据

A rationale from frequency perspective for grokking in training neural network

本文从频率视角实验揭示神经网络先学习弱显著频率成分，从而解释grokking现象，为理解其机制提供新视角。

rKAN：有理Kolmogorov-Arnold网络

rKAN: Rational Kolmogorov-Arnold Networks

针对KAN原有B样条基函数实现复杂的问题，研究提出以有理函数为新型基函数的rKAN，基于Pade逼近和有理雅可比函数构建可训练基函数，在多类任务中验证了其函数逼近的实用性与有效性。

Neural Network Processing Neural Networks: An efficient way to learn higher order functions

Functions are rich in meaning and can be interpreted in a variety of ways. Neural networks were proven to be capable of approximating a large class of functions[1]. In this paper, we propose a new class of neural networks called "Neural Network Processing Neural Networks" (NNPNNs), which inputs neural networks and numerical values, instead of just numerical values. Thus enabling neural networks to represent and process rich structures.

General Regression Neural Networks, Radial Basis Function Neural Networks, Support Vector Machines, and Feedforward Neural Networks

The aim of this project is to develop a code to discover the optimal sigma value that maximum the F1 score and the optimal sigma value that maximizes the accuracy and to find out if they are the same. Four algorithms which can be used to solve this problem are: Genetic Regression Neural Networks (GRNNs), Radial Based Function (RBF) Neural Networks (RBFNNs), Support Vector Machines (SVMs) and Feedforward Neural Network (FFNNs).