In this paper, we present a sharper version of the results in the paper Dimension independent bounds for general shallow networks; Neural Networks, \textbf{123} (2020), 142-152. Let $\mathbb{X}$ and $\mathbb{Y}$ be compact metric spaces. We consider approximation of functions of the form $ x\mapsto\int_{\mathbb{Y}} G( x, y)dτ( y)$, $ x\in\mathbb{X}$, by $G$-networks of the form $ x\mapsto \sum_{k=1}^n a_kG( x, y_k)$, $ y_1,\cdots, y_n\in\mathbb{Y}$, $a_1,\cdots, a_n\in\mathbb{R}$. Defining the dimensions of $\mathbb{X}$ and $\mathbb{Y}$ in terms of covering numbers, we obtain dimension independent bounds on the degree of approximation in terms of $n$, where also the constants involved are all dependent at most polynomially on the dimensions. Applications include approximation by power rectified linear unit networks, zonal function networks, certain radial basis function networks as well as the important problem of function extension to higher dimensional spaces.
Two aspects of neural networks that have been extensively studied in the recent literature are their function approximation properties and their training by gradient descent methods. The approximation problem seeks accurate approximations with a minimal number of weights. In most of the current literature these weights are fully or partially hand-crafted, showing the capabilities of neural networks but not necessarily their practical performance. In contrast, optimization theory for neural networks heavily relies on an abundance of weights in over-parametrized regimes. This paper balances thes…
We prove Carl's type inequalities for the error of approximation of compact sets K by deep and shallow neural networks. This in turn gives lower bounds on how well we can approximate the functions in K when requiring the approximants to come from outputs of such networks. Our results are obtained as a byproduct of the study of the recently introduced Lipschitz widths.
A physics informed neural network approach to simulating ice dynamics governed by the shallow ice approximation
Department of Mathematics and the Institute for Scientific Computing and Applied Mathematics, Indiana University（数学系和科学计算与应用数学研究所，印第安纳大学）；Department of Mathematics and Cognitive Science Program, Indiana University（数学系和认知科学项目，印第安纳大学）
SUPN: Shallow Universal Polynomial Networks for Efficient Function Approximation
Sandia National Laboratories（桑地亚国家实验室）；National Technology and Engineering Solutions of Sandia, LLC（桑地亚技术与工程解决方案公司）；Honeywell International Inc.（霍尼韦尔国际公司）
Gran Sasso Science Institute (GSSI)（格兰萨索科学研究所）；University of Cambridge（剑桥大学）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）；University of Edinburgh（爱丁堡大学）
We consider the approximation rates of shallow neural networks with respect to the variation norm. Upper bounds on these rates have been established for sigmoidal and ReLU activation functions, but it has remained an important open problem whether these rates are sharp. In this article, we provide a solution to this problem by proving sharp lower bounds on the approximation rates for shallow neural networks, which are obtained by lower bounding the $L^2$-metric entropy of the convex hull of the neural network basis functions. In addition, our methods also give sharp lower bounds on the Kolmogo…
Institute of Mathematical Sciences, Claremont Graduate University（克莱蒙特研究生大学数学科学研究所）
This paper proves an abstract theorem addressing in a unified manner two important problems in function approximation: avoiding curse of dimensionality and estimating the degree of approximation for out-of-sample extension in manifold learning. We consider an abstract (shallow) network that includes, for example, neural networks, radial basis function networks, and kernels on data defined manifolds used for function approximation in various settings. A deep network is obtained by a composition of the shallow networks according to a directed acyclic graph, representing the architecture of the d…
Center for Data Science, New York University（纽约大学数据科学中心）；Inria - Département d’Informatique de l’École Normale Supérieure（法国国家信息与自动化研究所 - 巴黎高等师范学校计算机科学系）；PSL Research University（巴黎文理研究大学）
Deep networks are often considered to be more expressive than shallow ones in terms of approximation. Indeed, certain functions can be approximated by deep networks provably more efficiently than by shallow ones, however, no tractable algorithms are known for learning such deep models. Separately, a recent line of work has shown that deep networks trained with gradient descent may behave like (tractable) kernel methods in a certain over-parameterized regime, where the kernel is determined by the architecture and initialization, and this paper focuses on approximation for such kernels. We show …
Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization
School of Mathematical Sciences Fudan University（复旦大学数学学院）；School of Mathematical Sciences Institute of Natural Sciences and MOE-LSC Shanghai Jiao Tong University（上海交通大学数学学院、自然科学研究院和教育部低维量子体系科学重点实验室）；School of Mathematical Sciences Shanghai Key Laboratory for Contemporary Applied Mathematics Fudan University（复旦大学上海当代应用数学重点实验室数学学院）
International School of Advanced Studies (SISSA)（国际高等研究学院（SISSA））；Duke University（杜克大学）；Université PSL（巴黎文理研究大学）；Sorbonne Université（索邦大学）；Université Paris-Diderot（巴黎第七大学）
Understanding the impact of data structure on the computational tractability of learning is a key challenge for the theory of neural networks. Many theoretical works do not explicitly model training data, or assume that inputs are drawn component-wise independently from some simple probability distribution. Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models. This is possible due to a Gaussian equivalence stating that the key metrics of interest, such as the training and test errors, can be fully captur…
Institute of Mathematical Sciences, Claremont Graduate University（克莱蒙特研究生大学数学科学研究所）；Center for Brains, Minds, and Machines, McGovern Institute for Brain Research, Massachusetts Institute of Technology（麻省理工学院麦戈文脑科学研究所脑、心智与机器中心）
We show that deep networks are better than shallow networks at approximating functions that can be expressed as a composition of functions described by a directed acyclic graph, because the deep networks can be designed to have the same compositional structure, while a shallow network cannot exploit this knowledge. Thus, the blessing of compositionality mitigates the curse of dimensionality. On the other hand, a theorem called good propagation of errors allows to `lift' theorems about shallow networks to those about deep networks with an appropriate choice of norms, smoothness, etc. We illustr…
We study the variation space corresponding to a dictionary of functions in $L^2(Ω)$ for a bounded domain $Ω\subset \mathbb{R}^d$. Specifically, we compare the variation space, which is defined in terms of a convex hull with related notions based on integral representations. This allows us to show that three important notions relating to the approximation theory of shallow neural networks, the Barron space, the spectral Barron space, and the Radon BV space, are actually variation spaces with respect to certain natural dictionaries.
University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper establishes rates of universal approximation for the shallow neural tangent kernel (NTK): network weights are only allowed microscopic changes from random initialization, which entails that activations are mostly unchanged, and the network is nearly equivalent to its linearization. Concretely, the paper has two main contributions: a generic scheme to approximate functions with the NTK by sampling from transport mappings between the initial weights and their desired values, and the construction of transport mappings via Fourier transforms. Regarding the first contribution, the proof …
TU München（慕尼黑工业大学）；Czech Technical University in Prague（捷克理工大学）；Duke University（杜克大学）
We address the structure identification and the uniform approximation of sums of ridge functions $f(x)=\sum_{i=1}^m g_i(a_i\cdot x)$ on ${\mathbb R}^d$, representing a general form of a shallow feed-forward neural network, from a small number of query samples. Higher order differentiation, as used in our constructive approximations, of sums of ridge functions or of their compositions, as in deeper neural network, yields a natural connection between neural network weight identification and tensor product decomposition identification. In the case of the shallowest feed-forward neural network, se…
Hong Kong Baptist University（香港浸会大学）；CityU（香港城市大学）
Based on the tree architecture, the objective of this paper is to design deep neural networks with two or more hidden layers (called deep nets) for realization of radial functions so as to enable rotational invariance for near-optimal function approximation in an arbitrarily high dimensional Euclidian space. It is shown that deep nets have much better performance than shallow nets (with only one hidden layer) in terms of approximation accuracy and learning capabilities. In particular, for learning radial functions, it is shown that near-optimal rate can be achieved by deep nets but not by shal…
