Lower bounds for artificial neural network approximations: A proof that shallow neural networks fail to overcome the curse of dimensionality
Faculty of Mathematics and Research Platform Data Science, University of Vienna（维也纳大学数学系与数据科学研究平台）；Faculty of Mathematics and Computer Science, University of Münster（明斯特大学数学与计算机科学学院）；School of Data Science and Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学与深圳大数据研究院）；Faculty of Mathematics, University of Vienna（维也纳大学数学系）
Philipp Grohs, Shokhrukh Ibragimov, Arnulf Jentzen, Sarah Koppensteiner
Artificial neural networks (ANNs) have become a very powerful tool in the approximation of high-dimensional functions. Especially, deep ANNs, consisting of a large number of hidden layers, have been very successfully used in a series of practical relevant computational problems involving high-dimensional input data ranging from classification tasks in supervised learning to optimal decision problems in reinforcement learning. There are also a number of mathematical results in the scientific literature which study the approximation capacities of ANNs in the context of high-dimensional target functions. In particular, there are a series of mathematical results in the scientific literature which show that sufficiently deep ANNs have the capacity to overcome the curse of dimensionality in the approximation of certain target function classes in the sense that the number of parameters of the approximating ANNs grows at most polynomially in the dimension $d \in \mathbb{N}$ of the target functions under considerations. In the proofs of several of such high-dimensional approximation results it is crucial that the involved ANNs are sufficiently deep and consist a sufficiently large number of hidden layers which grows in the dimension of the considered target functions. It is the topic of this work to look a bit more detailed to the deepness of the involved ANNs in the approximation of high-dimensional target functions. In particular, the main result of this work proves that there exists a concretely specified sequence of functions which can be approximated without the curse of dimensionality by sufficiently deep ANNs but which cannot be approximated without the curse of dimensionality if the involved ANNs are shallow or not deep enough.
We prove a theorem concerning the approximation of bandlimited multivariate functions by deep ReLU networks for which the curse of the dimensionality is overcome. Our theorem is based on a result by Maurey and on the ability of deep ReLU networks to approximate Chebyshev polynomials and analytic functions efficiently.
Full history recursive multilevel Picard approximations suffer from the curse of dimensionality for the Hamilton-Jacobi-Bellman equation of a stochastic control problem
We show that the minimal discrepancy of a point set in the $d$-dimensional unit cube with respect to the BMO seminorm suffers from the curse of dimensionality.
The recovery of ridge functions on the hypercube suffers from the curse of dimensionality
Laboratoire d’Informatique (LIX), École Polytechnique（École Polytechnique）；Fraunhofer Center for Machine Learning and Fraunhofer-Institute for Algorithms and Scientific Computing SCAI（弗劳恩霍夫机器学习中心与算法和科学计算研究所 SCAI）
Recently, so-called full-history recursive multilevel Picard (MLP) approximation schemes have been introduced and shown to overcome the curse of dimensionality in the numerical approximation of semilinear parabolic partial differential equations (PDEs) with Lipschitz nonlinearities. The key contribution of this article is to introduce and analyze a new variant of MLP approximation schemes for certain semilinear elliptic PDEs with Lipschitz nonlinearities and to prove that the proposed approximation schemes overcome the curse of dimensionality in the numerical approximation of such semilinear e…
An algorithm (CoDeFi) for overcoming the curse of dimensionality in mathematical finance
Laboratoire Jacques-Louis Lions & Centre National de la Recherche Scientifique Université Pierre et Marie Curie（雅克-路易·利翁斯实验室 & 法国国家科学研究中心 皮埃尔和玛丽·居里大学）；MPG-Partners
In this paper, we analyze the number of neurons and training parameters that a neural networks needs to approximate multivariate functions of bounded second mixed derivatives -- Korobov functions. We prove upper bounds on these quantities for shallow and deep neural networks, breaking the curse of dimensionality. Our bounds hold for general activation functions, including ReLU. We further prove that these bounds nearly match the minimal number of parameters any continuous function approximator needs to approximate Korobov functions, showing that neural networks are near-optimal function approx…
Deep neural networks and other deep learning methods have very successfully been applied to the numerical approximation of high-dimensional nonlinear parabolic partial differential equations (PDEs), which are widely used in finance, engineering, and natural sciences. In particular, simulations indicate that algorithms based on deep learning overcome the curse of dimensionality in the numerical approximation of solutions of semilinear PDEs. For certain linear PDEs this has also been proved mathematically. The key contribution of this article is to rigorously prove this for the first time for a …
We prove that deep neural networks are capable of approximating solutions of semilinear Kolmogorov PDE in the case of gradient-independent, Lipschitz-continuous nonlinearities, while the required number of parameters in the networks grow at most polynomially in both dimension $d \in \mathbb{N}$ and prescribed reciprocal accuracy $\varepsilon$. Previously, this has only been proven in the case of semilinear heat equations.
Partial differential equations (PDEs) are a fundamental tool in the modeling of many real world phenomena. In a number of such real world phenomena the PDEs under consideration contain gradient-dependent nonlinearities and are high-dimensional. Such high-dimensional nonlinear PDEs can in nearly all cases not be solved explicitly and it is one of the most challenging tasks in applied mathematics to solve high-dimensional nonlinear PDEs approximately. It is especially very challenging to design approximation algorithms for nonlinear PDEs for which one can rigorously prove that they do overcome t…
Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces. The approximation error of the neural network is $O(1/\sqrt{m})$ where $m$ is the size of networks, which overcomes the curse of dimensionality. The key idea of the approximation is to define a Barron spectral space of functionals.
One of the most challenging problems in applied mathematics is the approximate solution of nonlinear partial differential equations (PDEs) in high dimensions. Standard deterministic approximation methods like finite differences or finite elements suffer from the curse of dimensionality in the sense that the computational effort grows exponentially in the dimension. In this work we overcome this difficulty in the case of reaction-diffusion type PDEs with a locally Lipschitz continuous coervice nonlinearity (such as Allen-Cahn PDEs) by introducing and analyzing truncated variants of the recently…
The hierarchical (multi-linear) rank of an order-$d$ tensor is key in determining the cost of representing a tensor as a (tree) Tensor Network (TN). In general, it is known that, for a fixed accuracy, a tensor with random entries cannot be expected to be efficiently approximable without the curse of dimensionality, i.e., a complexity growing exponentially with $d$. In this work, we show that the ground state projection (GSP) of a class of unbounded Hamiltonians can be approximately represented as an operator of low effective dimensionality that is independent of the (high) dimension $d$ of the…
We propose a deep neural network architecture for storing approximate Lyapunov functions of systems of ordinary differential equations. Under a small-gain condition on the system, the number of neurons needed for an approximation of a Lyapunov function with fixed accuracy grows only polynomially in the state dimension, i.e., the proposed approach is able to overcome the curse of dimensionality.
For a long time it is well-known that high-dimensional linear parabolic partial differential equations (PDEs) can be approximated by Monte Carlo methods with a computational effort which grows polynomially both in the dimension and in the reciprocal of the prescribed accuracy. In other words, linear PDEs do not suffer from the curse of dimensionality. For general semilinear PDEs with Lipschitz coefficients, however, it remained an open question whether these suffer from the curse of dimensionality. In this paper we partially solve this open problem. More precisely, we prove in the case of semi…
Multilevel Picard approximations overcome the curse of dimensionality when approximating semilinear heat equations with gradient-dependent nonlinearities in $L^p$-sense
Backward stochastic differential equations (BSDEs) belong nowadays to the most frequently studied equations in stochastic analysis and computational stochastics. BSDEs in applications are often nonlinear and high-dimensional. In nearly all cases such nonlinear high-dimensional BSDEs cannot be solved explicitly and it has been and still is a very active topic of research to design and analyze numerical approximation methods to approximatively solve nonlinear high-dimensional BSDEs. Although there are a large number of research articles in the scientific literature which analyze numerical approx…
Multilevel Picard approximations overcome the curse of dimensionality in the numerical approximation of general semilinear PDEs with gradient-dependent nonlinearities
In this paper, we establish that for a wide class of controlled stochastic differential equations (SDEs) with stiff coefficients, the value functions of corresponding zero-sum games can be represented by a deep artificial neural network (DNN), whose complexity grows at most polynomially in both the dimension of the state equation and the reciprocal of the required accuracy. Such nonlinear stiff systems may arise, for example, from Galerkin approximations of controlled stochastic partial differential equations (SPDEs), or controlled PDEs with uncertain initial conditions and source terms. This …
Deep neural networks (DNNs) with ReLU activation function are proved to be able to express viscosity solutions of linear partial integrodifferental equations (PIDEs) on state spaces of possibly high dimension $d$. Admissible PIDEs comprise Kolmogorov equations for high-dimensional diffusion, advection, and for pure jump Lévy processes. We prove for such PIDEs arising from a class of jump-diffusions on $\mathbb{R}^d$ that for any suitable measure $μ^d$ on $\mathbb{R}^d$ there exist constants $C,{\mathfrak{p}},{\mathfrak{q}}>0$ such that for every $\varepsilon \in (0,1]$ and for every $d\in \mat…
Recently, it has been proposed in the literature to employ deep neural networks (DNNs) together with stochastic gradient descent methods to approximate solutions of PDEs. There are also a few results in the literature which prove that DNNs can approximate solutions of certain PDEs without the curse of dimensionality in the sense that the number of real parameters used to describe the DNN grows at most polynomially both in the PDE dimension and the reciprocal of the prescribed approximation accuracy. One key argument in most of these results is, first, to use a Monte Carlo approximation scheme …
University of Vienna（维也纳大学）；ETH Zürich（苏黎世联邦理工学院）；University of Münster（明斯特大学）
The development of new classification and regression algorithms based on empirical risk minimization (ERM) over deep neural network hypothesis classes, coined deep learning, revolutionized the area of artificial intelligence, machine learning, and data analysis. In particular, these methods have been applied to the numerical solution of high-dimensional partial differential equations with great success. Recent simulations indicate that deep learning-based algorithms are capable of overcoming the curse of dimensionality for the numerical solution of Kolmogorov equations, which are widely used i…
Multilevel Picard approximations and deep neural networks with ReLU, leaky ReLU, and softplus activation overcome the curse of dimensionality when approximating semilinear parabolic partial differential equations in $L^p$-sense
Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore（数学科学系，物理与数学科学学院，南洋理工大学，新加坡）；Faculty of Mathematics, Bielefeld University, Germany（数学学院，比勒菲尔德大学，德国）
Breaking the Curse of Dimensionality in Quantum PDE Solvers via Gevrey Regularity
University of Waterloo（滑铁卢大学）；Perimeter Institute for Theoretical Physics（理论物理前沿研究所）；Microsoft Quantum（微软量子）；Massachusetts Institute of Technology（麻省理工学院）；University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；National Research Council Canada（加拿大国家研究委员会）
Imperial College London（帝国理工学院）；University of Münster（明斯特大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
In this article we study high-dimensional approximation capacities of shallow and deep artificial neural networks (ANNs) with the rectified linear unit (ReLU) activation. In particular, it is a key contribution of this work to reveal that for all $a,b\in\mathbb{R}$ with $b-a\geq 7$ we have that the functions $[a,b]^d\ni x=(x_1,\dots,x_d)\mapsto\prod_{i=1}^d x_i\in\mathbb{R}$ for $d\in\mathbb{N}$ as well as the functions $[a,b]^d\ni x =(x_1,\dots, x_d)\mapsto\sin(\prod_{i=1}^d x_i) \in \mathbb{R} $ for $ d \in \mathbb{N} $ can neither be approximated without the curse of dimensionality by means…
In this paper, we construct neural networks with ReLU, sine and $2^x$ as activation functions. For general continuous $f$ defined on $[0,1]^d$ with continuity modulus $ω_f(\cdot)$, we construct ReLU-sine-$2^x$ networks that enjoy an approximation rate $\mathcal{O}(ω_f(\sqrt{d})\cdot2^{-M}+ω_{f}\left(\frac{\sqrt{d}}{N}\right))$, where $M,N\in \mathbb{N}^{+}$ denote the hyperparameters related to widths of the networks. As a consequence, we can construct ReLU-sine-$2^x$ network with the depth $5$ and width $\max\left\{\left\lceil2d^{3/2}\left(\frac{3μ}ε\right)^{1/α}\right\rceil,2\left\lceil\log_…
In recent years deep artificial neural networks (DNNs) have been successfully employed in numerical simulations for a multitude of computational problems including, for example, object and face recognition, natural language processing, fraud detection, computational advertisement, and numerical approximations of partial differential equations (PDEs). These numerical simulations indicate that DNNs seem to possess the fundamental flexibility to overcome the curse of dimensionality in the sense that the number of real parameters used to describe the DNN grows at most polynomially in both the reci…
Physics Informed Neural Networks (PINNs) have recently gained popularity for solving partial differential equations, given the fact they escape the curse of dimensionality. In this paper, we present Physics Informed Neural Networks as an underdetermined point matching collocation method then expose the connection between Galerkin Least Square (GALS) and PINNs, to develop an a priori error estimate, in the context of elliptic problems. In particular, techniques that belong to the realm of least square finite elements and Rademacher complexity analysis are used to obtain the error estimate.
A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
University of Vienna（维也纳大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；ETH Zurich（苏黎世联邦理工学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Münster（明斯特大学）
Space-time deep neural network approximations for high-dimensional partial differential equations
Faculty of Mathematics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院数学学院）；Seminar for Applied Mathematics, Department of Mathematics, ETH Zurich（苏黎世联邦理工学院数学系应用数学研讨班）；Department for Applied Mathematics, University of Freiburg（弗莱堡大学应用数学系）
Tackling the Curse of Dimensionality in Fractional and Tempered Fractional PDEs with Physics-Informed Neural Networks
National University of Singapore（新加坡国立大学）；Worcester Polytechnic Institute（伍斯特理工学院）；Brown University（布朗大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）
本文将 MC-fPINN 扩展为 MC-tfPINN，并用一维 Gaussian 求积替代 Monte Carlo 采样，在高达 100,000 维的分数阶与缓增分数阶 PDE 正、反问题中提升了精度和收敛速度。
Backward stochastic differential equations (BSDEs) appear in numeruous applications. Classical approximation methods suffer from the curse of dimensionality and deep learning-based approximation methods are not known to converge to the BSDE solution. Recently, Hutzenthaler et al. (arXiv:2108.10602) introduced a new approximation method for BSDEs whose forward diffusion is Brownian motion and proved that this method converges with essentially optimal rate without suffering from the curse of dimensionality. The central object of this article is to extend this result to general forward diffusions…
