We perform a comprehensive numerical study of the effect of approximation-theoretical results for neural networks on practical learning problems in the context of numerical analysis. As the underlying model, we study the machine-learning-based solution of parametric partial differential equations. Here, approximation theory predicts that the performance of the model should depend only very mildly on the dimension of the parameter space and is determined by the intrinsic dimension of the solution manifold of the parametric partial differential equation. We use various methods to establish comparability between test-cases by minimizing the effect of the choice of test-cases on the optimization and sampling aspects of the learning problem. We find strong support for the hypothesis that approximation-theoretical effects heavily influence the practical behavior of learning problems in numerical analysis.
The Deep Operator Networks~(DeepONet) is a fundamentally different class of neural networks that we train to approximate nonlinear operators, including the solution operator of parametric partial differential equations (PDE). DeepONets have shown remarkable approximation and generalization capabilities even when trained with relatively small datasets. However, the performance of DeepONets deteriorates when the training data is polluted with noise, a scenario that occurs very often in practice. To enable DeepONets training with noisy data, we propose using the Bayesian framework of replica-exch…
Oak Ridge National Laboratory（橡树岭国家实验室）；Politecnico di Milano（米兰理工大学）
We propose a new approach to generate a reliable reduced model for a parametric elliptic problem, in the presence of noisy data. The reference model reduction procedure is the directional HiPOD method, which combines Hierarchical Model reduction with a standard Proper Orthogonal Decomposition, according to an offline/online paradigm. In this paper we show that directional HiPOD looses in terms of accuracy when problem data are affected by noise. This is due to the interpolation driving the online phase, since it replicates, by definition, the noise trend. To overcome this limit, we replace int…
For the evolution of a closed surface under anisotropic surface diffusion with a general anisotropic surface energy $γ(\boldsymbol{n})$ in three dimensions (3D), where $\boldsymbol{n}$ is the unit outward normal vector, by introducing a novel symmetric positive definite surface energy matrix $\boldsymbol{Z}_k(\boldsymbol{n})$ depending on a stabilizing function $k(\boldsymbol{n})$ and the Cahn-Hoffman $\boldsymbolξ$-vector, we present a new symmetrized variational formulation for anisotropic surface diffusion with weakly or strongly anisotropic surface energy, which preserves two important str…
We propose a structure-preserving parametric finite element method (SP-PFEM) for discretizing the surface diffusion of a closed curve in two dimensions (2D) or surface in three dimensions (3D). Here the "structure-preserving" refers to preserving the two fundamental geometric structures of the surface diffusion flow: (i) the conservation of the area/volume enclosed by the closed curve/surface, and (ii) the decrease of the perimeter/total surface area of the curve/surface. For simplicity of notations, we begin with the surface diffusion of a closed curve in 2D and present a weak (variational) f…
We propose an energy-stable parametric finite element method (ES-PFEM) to discretize the motion of a closed curve under surface diffusion with an anisotropic surface energy $γ(θ)$ -- anisotropic surface diffusion -- in two dimensions, while $θ$ is the angle between the outward unit normal vector and the vertical axis. By introducing a positive definite surface energy (density) matrix $G(θ)$, we present a new and simple variational formulation for the anisotropic surface diffusion and prove that it satisfies area/mass conservation and energy dissipation. The variational problem is discretized i…
Technische Universität Berlin（柏林工业大学）；University of Tromsø（特罗姆瑟大学）；University of Vienna（维也纳大学）
We derive upper bounds on the complexity of ReLU neural networks approximating the solution maps of parametric partial differential equations. In particular, without any knowledge of its concrete shape, we use the inherent low-dimensionality of the solution manifold to obtain approximation rates which are significantly superior to those provided by classical neural network approximation results. Concretely, we use the existence of a small reduced basis to construct, for a large variety of parametric partial differential equations, neural networks that yield approximations of the parametric sol…
University of Freiburg（弗莱堡大学）；Simula Research Laboratory（西穆拉研究实验室）
We establish error estimates for the approximation of parametric $p$-Dirichlet problems deploying the Deep Ritz Method. Parametric dependencies include, e.g., varying geometries and exponents $p\in (1,\infty)$. Combining the derived error estimates with quantitative approximation theorems yields error decay rates and establishes that the Deep Ritz Method retains the favorable approximation capabilities of neural networks in the approximation of high dimensional functions which makes the method attractive for parametric problems. Finally, we present numerical examples to illustrate potential ap…
We derive rigorous bounds on the error resulting from the approximation of the solution of parametric hyperbolic scalar conservation laws with ReLU neural networks. We show that the approximation error can be made as small as desired with ReLU neural networks that overcome the curse of dimensionality. In addition, we provide an explicit upper bound on the generalization error in terms of the training error, number of training samples and the neural network size. The theoretical results are illustrated by numerical experiments.
University of Vienna（维也纳大学）；University of Oxford（牛津大学）；Austrian Academy of Sciences（奥地利科学院）
We present a deep learning algorithm for the numerical solution of parametric families of high-dimensional linear Kolmogorov partial differential equations (PDEs). Our method is based on reformulating the numerical approximation of a whole family of Kolmogorov PDEs as a single statistical learning problem using the Feynman-Kac formula. Successful numerical experiments are presented, which empirically confirm the functionality and efficiency of our proposed algorithm in the case of heat equations and Black-Scholes option pricing models parametrized by affine-linear coefficient functions. We sho…
Generative diffusion learning for parametric partial differential equations
Booz Allen Hamilton Inc（博思艾伦咨询公司）；Physical Modeling and Simulation Branch, DEVCOM Army Research Laboratory（DEVCOM陆军研究实验室物理建模与模拟分支）；Department of Mathematical Sciences, University of Delaware（特拉华大学数学科学系）
Implementing deep neural networks for learning the solution maps of parametric partial differential equations (PDEs) turns out to be more efficient than using many conventional numerical methods. However, limited theoretical analyses have been conducted on this approach. In this study, we investigate the expressive power of deep rectified quadratic unit (ReQU) neural networks for approximating the solution maps of parametric PDEs. The proposed approach is motivated by the recent important work of G. Kutyniok, P. Petersen, M. Raslan and R. Schneider (Gitta Kutyniok, Philipp Petersen, Mones Rasl…
Deep operator networks (DeepONets) are receiving increased attention thanks to their demonstrated capability to approximate nonlinear operators between infinite-dimensional Banach spaces. However, despite their remarkable early promise, they typically require large training data-sets consisting of paired input-output observations which may be expensive to obtain, while their predictions may not be consistent with the underlying physical principles that generated the observed data. In this work, we propose a novel model class coined as physics-informed DeepONets, which introduces an effective r…
We deal with a long-standing problem about how to design an energy-stable numerical scheme for solving the motion of a closed curve under {\sl anisotropic surface diffusion} with a general anisotropic surface energy $γ(\boldsymbol{n})$ in two dimensions, where $\boldsymbol{n}$ is the outward unit normal vector. By introducing a novel symmetric positive definite surface energy matrix $Z_k(\boldsymbol{n})$ depending on the Cahn-Hoffman $\boldsymbolξ$-vector and a stabilizing function $k(\boldsymbol{n})$, we first reformulate the anisotropic surface diffusion into a conservative form and then der…
Deep Eigenspace Network for Parametric Non-self-adjoint Eigenvalue Problems
Department of Mathematics, The University of Hong Kong（香港大学数学系）；Department of Mathematical Sciences, Michigan Technological University（密歇根理工大学数学科学系）；Materials Innovation Institute for Life Sciences and Energy (MILES), HKU-SIRI（生命科学与能源材料创新研究院（MILES），HKU-SIRI）
SISSA（的里雅斯特国际高等研究院）；University of Urbino（乌尔比诺大学）；University of Urbino Carlo Bo（卡洛·博乌尔比诺大学）
Parametric time-dependent systems are of a crucial importance in modeling real phenomena, often characterized by non-linear behaviors too. Those solutions are typically difficult to generalize in a sufficiently wide parameter space while counting on limited computational resources available. As such, we present a general two-stages deep learning framework able to perform that generalization with low computational effort in time. It consists in a separated training of two pipe-lined predictive models. At first, a certain number of independent neural networks are trained with data-sets taken fro…
We investigate non-adaptive methods of deep ReLU neural network approximation in Bochner spaces $L_2({\mathbb U}^\infty, X, μ)$ of functions on ${\mathbb U}^\infty$ taking values in a separable Hilbert space $X$, where ${\mathbb U}^\infty$ is either ${\mathbb R}^\infty$ equipped with the standard Gaussian probability measure, or ${\mathbb I}^\infty:= [-1,1]^\infty$ equipped with the Jacobi probability measure. Functions to be approximated are assumed to satisfy a certain weighted $\ell_2$-summability of the generalized chaos polynomial expansion coefficients with respect to the measure $μ$. We…
The Alan Turing Institute（艾伦·图灵研究所）；University of Edinburgh（爱丁堡大学）；Vega
We develop several deep learning algorithms for approximating families of parametric PDE solutions. The proposed algorithms approximate solutions together with their gradients, which in the context of mathematical finance means that the derivative prices and hedging strategies are computed simulatenously. Having approximated the gradient of the solution one can combine it with a Monte-Carlo simulation to remove the bias in the deep network approximation of the PDE solution (derivative price). This is achieved by leveraging the Martingale Representation Theorem and combining the Monte Carlo sim…
In this paper, we propose a network model, the multiclass classification-based reduced order model (MC-ROM), for solving time-dependent parametric partial differential equations (PPDEs). This work is inspired by the observation of applying the deep learning-based reduced order model (DL-ROM) to solve diffusion-dominant PPDEs. We find that the DL-ROM has a good approximation for some special model parameters, but it cannot approximate the drastic changes of the solution as time evolves. Based on this fact, we classify the dataset according to the magnitude of the solutions and construct corresp…
Stochastic Galerkin finite element method (SGFEM) provides an efficient alternative to traditional sampling methods for the numerical solution of linear elliptic partial differential equations with parametric or random inputs. However, computing stochastic Galerkin approximations for a given problem requires the solution of large coupled systems of linear equations. Therefore, an effective and bespoke iterative solver is a key ingredient of any SGFEM implementation. In this paper, we analyze a class of truncation preconditioners for SGFEM. Extending the idea of the mean-based preconditioner, t…
We obtained convergence rates of the collocation approximation by deep ReLU neural networks of solutions to elliptic PDEs with lognormal inputs, parametrized by $\boldsymbol{y}$ from the non-compact set $\mathbb{R}^\infty$. The approximation error is measured in the norm of the Bochner space $L_2(\mathbb{R}^\infty, V, γ)$, where $γ$ is the infinite tensor product standard Gaussian probability measure on $\mathbb{R}^\infty$ and $V$ is the energy space. We also obtained similar results for the case when the lognormal inputs are parametrized on $\mathbb{R}^M$ with very large dimension $M$, and th…
We demonstrate that deep neural networks with the ReLU activation function can efficiently approximate the solutions of various types of parametric linear transport equations. For non-smooth initial conditions, the solutions of these PDEs are high-dimensional and non-smooth. Therefore, approximation of these functions suffers from a curse of dimension. We demonstrate that through their inherent compositionality deep neural networks can resolve the characteristic flow underlying the transport equations and thereby allow approximation rates independent of the parameter dimension.
