We consider the problem of univariate nonlinear function approximation using shallow neural networks (NN) with a rectified linear unit (ReLU) activation function. We show that the $L_2$ based approximation problem is ill-conditioned and the behaviour of optimisation algorithms used in training these networks degrades rapidly as the width of the network increases. This can lead to significantly poorer approximation in practice than expected from the theoretical expressivity of the ReLU architecture and traditional methods such as univariate Free Knot Splines (FKS). Univariate shallow ReLU NNs and FKS span the same function space, and thus have the same theoretical expressivity. However, the FKS representation remains well-conditioned as the number of knots increases. We leverage the theory of optimal piecewise linear interpolants to improve the training procedure for ReLU NNs. Using the equidistribution principle, we propose a two-level procedure for training the FKS by first solving the nonlinear problem of finding the optimal knot locations of the interpolating FKS, and then determine the optimal weights and knots of the FKS by solving a nearly linear, well-conditioned problem. The training of the FKS gives insights into how we can train a ReLU NN effectively, with an equally accurate approximation. We combine the training of the ReLU NN with an equidistribution-based loss to find the breakpoints of the ReLU functions. This is then combined with preconditioning the ReLU NN approximation to find the scalings of the ReLU functions. This fast, well-conditioned and reliable method finds an accurate shallow ReLU NN approximation to a univariate target function. We test this method on a series of regular, singular, and rapidly varying target functions and obtain good results, realising the expressivity of the shallow ReLU network in all cases. We then extend our results to deeper networks.
已经为神经网络提出了各种各样的激活函数。Rectified Linear Unit（ReLU）是一个非常流行的算法。有许多实际原因促使使用ReLU。本文提供了新的理论特征，支持使用ReLU，其变体（如Leaky ReLU）以及单变量单隐层前馈神经网络情况下的其他激活函数。我们的研究结果还解释了在神经网络的设计和训练中常用策略的重要性，如“权重衰减”和“路径范数”正则化，并为在网络架构中使用“跳过连接”提供了新的理由。这些新的见解是通过样条理论的透镜获得的。特别是，我们展示了如何神经网络训练问题是有关的Banach空间的函数，其解决方案是众所周知的分数和多项式样条，其中特定的Banach空间（控制的阶样条）取决于激活函数的选择的无限维优化。
Multilevel Training for Kolmogorov Arnold Networks
Theoretical Division, Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室理论 division）；Center for Computing Research, Sandia National Laboratories（桑迪亚国家实验室计算研究中心）
How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer
ETH Z\"urich, D-Math, R\"amistrasse 101, CH-8092 Z\"urich, Switzerland
ReLU-KAN: New Kolmogorov-Arnold Networks that Only Need Matrix Addition, Dot Multiplication, and ReLU
University of South China（南华大学）；Paris Elite Institute of Technology, Shanghai Jiao Tong University（上海交通大学巴黎卓越工程师学院）；University of Science and Technology Beijing（北京科技大学）；Dalian University of Technology（大连理工大学）
针对 KAN 因 B 样条基函数导致 GPU 并行效率低的问题，本文提出仅用 ReLU 与逐点乘法的 ReLU-KAN，在 4 层网络上较传统 KAN 提速 20 倍，并保持更稳定训练、更强拟合及抗遗忘特性。
