This paper aims at studying the difference between Ritz-Galerkin (R-G) method and deep neural network (DNN) method in solving partial differential equations (PDEs) to better understand deep learning. To this end, we consider solving a particular Poisson problem, where the information of the right-hand side of the equation f is only available at n sample points, that is, f is known at finite sample points. Through both theoretical and numerical studies, we show that solution of the R-G method converges to a piecewise linear function for the one dimensional (1D) problem or functions of lower regularity for high dimensional problems. With the same setting, DNNs however learn a relative smooth solution regardless of the dimension, this is, DNNs implicitly bias towards functions with more low-frequency components among all functions that can fit the equation at available data points. This bias is explained by the recent study of frequency principle (Xu et al., (2019) [17] and Zhang et al., (2019) [11, 19]). In addition to the similarity between the traditional numerical methods and DNNs in the approximation perspective, our work shows that the implicit bias in the learning process, which is different from traditional numerical methods, could help better understand the characteristics of DNNs.
Recent years have witnessed growing interests in solving partial differential equations by deep neural networks, especially in the high-dimensional case. Unlike classical numerical methods, such as finite difference method and finite element method, the enforcement of boundary conditions in deep neural networks is highly nontrivial. One general strategy is to use the penalty method. In the work, we conduct a comparison study for elliptic problems with four different boundary conditions, i.e., Dirichlet, Neumann, Robin, and periodic boundary conditions, using two representative methods: deep Ga…
In this paper, we study adaptive neuron enhancement (ANE) method for solving self-adjoint second-order elliptic partial differential equations (PDEs). The ANE method is a self-adaptive method generating a two-layer spline NN and a numerical integration mesh such that the approximation accuracy is within the prescribed tolerance. Moreover, the ANE method provides a natural process for obtaining a good initialization which is crucial for training nonlinear optimization problem. The underlying PDE is discretized by the Ritz method using a two-layer spline neural network based on either the primal…
In this paper, we propose a Galerkin finite element method for the elliptic optimal control problem governed by the Riesz space-fractional PDEs on 2D domains with control variable being discretized by variational discretization technique. The optimality condition is derived and priori error estimates of control, costate and state variables are successfully established. Numerical test is carried out to illustrate the accuracy performance of this approach.
Using deep neural networks to solve PDEs has attracted a lot of attentions recently. However, why the deep learning method works is falling far behind its empirical success. In this paper, we provide a rigorous numerical analysis on deep Ritz method (DRM) \cite{Weinan2017The} for second order elliptic equations with Drichilet, Neumann and Robin boundary condition, respectively. We establish the first nonasymptotic convergence rate in $H^1$ norm for DRM using deep networks with smooth activation functions including logistic and hyperbolic tangent functions. Our results show how to set the hyper…
An adaptive Deep Ritz framework for second-order fully nonlinear partial differential equations
Geneva School of Business Administration (HEG-Genève), University of Applied Sciences and Arts Western Switzerland (HES-SO)（日内瓦商业管理学院（HEG-Genève），瑞士西部应用科学和艺术大学（HES-SO））；Institute of Mathematics, Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院数学研究所）
In this paper, we study the stabilizer-free weak Galerkin methods on polytopal meshes for a class of second order elliptic boundary value problems of divergence form and with gradient nonlinearity in the principal coefficient. With certain assumptions on the nonlinear coefficient, we show that the discrete problem has a unique solution. This is achieved by showing that the associated operator satisfies certain continuity and monotonicity properties. With the help of these properties, we derive optimal error estimates in the energy norm. We present several numerical examples to verify the error…
RUNNs: Ritz-Uzawa Neural Networks for Solving Variational Problems
Basque Center for Applied Mathematics（巴斯克应用数学中心）；Instituto de Matemáticas, Pontificia Universidad Católica de Valparaíso（天主教法科大学瓦帕莱索数学研究所）；Department of Mathematics, CUNEF Universidad（CUNEF大学数学系）；Curtin University（科廷大学）；Department of Mathematics, University of the Basque Country UPV/EHU（巴斯克国立大学数学系）；Ikerbasque: Basque Foundation for Science（伊克拉巴斯克：巴斯克科学基金会）；School of Mathematical Sciences, University of Nottingham（诺丁汉大学数学科学学院）
In this paper, we develop a new discontinuous Galerkin method for solving several types of partial differential equations (PDEs) with high order spatial derivatives. We combine the advantages of local discontinuous Galerkin (LDG) method and ultra-weak discontinuous Galerkin (UWDG) method. Firstly, we rewrite the PDEs with high order spatial derivatives into a lower order system, then apply the UWDG method to the system. We first consider the fourth order and fifth order nonlinear PDEs in one space dimension, and then extend our method to general high order problems and two space dimensions. Th…
Duke University（杜克大学）；University of Massachusetts（马萨诸塞大学）
This paper concerns the a priori generalization analysis of the Deep Ritz Method (DRM) [W. E and B. Yu, 2017], a popular neural-network-based method for solving high dimensional partial differential equations. We derive the generalization error bounds of two-layer neural networks in the framework of the DRM for solving two prototype elliptic PDEs: Poisson equation and static Schrödinger equation on the $d$-dimensional unit hypercube. Specifically, we prove that the convergence rates of generalization errors are independent of the dimension $d$, under the a priori assumption that the exact solu…
In this paper, we propose a fast spectral-Galerkin method for solving PDEs involving integral fractional Laplacian in $\mathbb{R}^d$, which is built upon two essential components: (i) the Dunford-Taylor formulation of the fractional Laplacian; and (ii) Fourier-like bi-orthogonal mapped Chebyshev functions (MCFs) as basis functions. As a result, the fractional Laplacian can be fully diagonalised, and the complexity of solving an elliptic fractional PDE is quasi-optimal, i.e., $O((N\log_2N)^d)$ with $N$ being the number of modes in each spatial direction. Ample numerical tests for various decayi…
We consider a linear elliptic partial differential equation (PDE) with a generic uniformly bounded parametric coefficient. The solution to this PDE problem is approximated in the framework of stochastic Galerkin finite element methods. We perform a posteriori error analysis of Galerkin approximations and derive a reliable and efficient estimate for the energy error in these approximations. Practical versions of this error estimate are discussed and tested numerically for a model problem with non-affine parametric representation of the coefficient. Furthermore, we use the error reduction indica…
While deep learning has achieved remarkable success in solving partial differential equations (PDEs), it still faces significant challenges, particularly when the PDE solutions have low regularity or singularities. To address these issues, we propose the Weak TransNet (WTN) method, based on a Petrov-Galerkin formulation, for solving elliptic PDEs in this work, though its framework may extend to other classes of equations. Specifically, the neural feature space defined by TransNet (Zhang et al., 2023) is used as the trial space, while the test space is composed of radial basis functions. Since …
Using deep neural networks to solve PDEs has attracted a lot of attentions recently. However, why the deep learning method works is falling far behind its empirical success. In this paper, we provide a rigorous numerical analysis on deep Ritz method (DRM) \cite{wan11} for second order elliptic equations with Neumann boundary conditions. We establish the first nonasymptotic convergence rate in $H^1$ norm for DRM using deep networks with $\mathrm{ReLU}^2$ activation functions. In addition to providing a theoretical justification of DRM, our study also shed light on how to set the hyper-parameter…
In this work, we derive a priori error estimate of the mixed residual method when solving some elliptic PDEs. Our work is the first theoretical study of this method. We prove that the neural network solutions will converge if we increase the training samples and network size without any constraint on the ratio of training samples to the network size. Besides, our results suggest that the mixed residual method can recover high order derivatives better than the deep Ritz method, which has also been verified by our numerical experiments.
Max Planck Institute for Mathematics in the Sciences（马克斯·普朗克数学研究所）；University of Freiburg（弗莱堡大学）
Recently, progress has been made in the application of neural networks to the numerical analysis of partial differential equations (PDEs). In the latter the variational formulation of the Poisson problem is used in order to obtain an objective function - a regularised Dirichlet energy - that was used for the optimisation of some neural networks. In this notes we use the notion of $Γ$-convergence to show that ReLU networks of growing architecture that are trained with respect to suitably regularised Dirichlet energies converge to the true solution of the Poisson problem. We discuss how this app…
The quantification of multivariate uncertainties in partial differential equations can easily exceed any computing capacity unless proper measures are taken to reduce the complexity of the model. In this work, we propose a multidimensional Galerkin Proper Orthogonal Decomposition that optimally reduces each dimension of a tensorized product space. We provide the analytical framework and results that define and quantify the low-dimensional approximation. We illustrate its application for uncertainty modeling with Polynomial Chaos Expansions and show its efficiency in a numerical example.
In this paper, we perform the convergence analysis of unsupervised Legendre--Galerkin neural networks (ULGNet), a deep-learning-based numerical method for solving partial differential equations (PDEs). Unlike existing deep learning-based numerical methods for PDEs, the ULGNet expresses the solution as a spectral expansion with respect to the Legendre basis and predicts the coefficients with deep neural networks by solving a variational residual minimization problem. Since the corresponding loss function is equivalent to the residual induced by the linear algebraic system depending on the choic…
This article considers the extension of two-grid $hp$-version discontinuous Galerkin finite element methods for the numerical approximation of second-order quasilinear elliptic boundary value problems of monotone type to the case when agglomerated polygonal/polyhedral meshes are employed for the coarse mesh approximation. We recall that within the two-grid setting, while it is necessary to solve a nonlinear problem on the coarse approximation space, only a linear problem must be computed on the original fine finite element space. In this article, the coarse space will be constructed by agglome…
Wavelet Galerkin method for fractional elliptic differential equations
School of Mathematics and Statistics, Gansu Key Laboratory of Applied Mathematics and Complex Systems, Lanzhou University（兰州大学数学与统计学院，甘肃省应用数学与复杂系统重点实验室）
We consider the time discretization of a linear parabolic problem by the discontinuous Galerkin (DG) method using piecewise polynomials of degree at most $r-1$ in $t$, for $r\ge1$ and with maximum step size~$k$. It is well known that the spatial $L_2$-norm of the DG error is of optimal order $k^r$ globally in time, and is, for $r\ge2$, superconvergent of order $k^{2r-1}$ at the nodes. We show that on the $n$th subinterval $(t_{n-1},t_n)$, the dominant term in the DG error is proportional to the local right Radau polynomial of degree $r$. This error profile implies that the DG error is of order…
The aim of this paper is to apply a high-order discontinuous-in-time scheme to second-order hyperbolic partial differential equations (PDEs). We first discretize the PDEs in time while keeping the spatial differential operators undiscretized. The well-posedness of this semi-discrete scheme is analyzed and a priori error estimates are derived in the energy norm. We then combine this $hp$-version discontinuous Galerkin method for temporal discretization with an $H^1$-conforming finite element approximation for the spatial variables to construct a fully discrete scheme. A prior error estimates ar…
Fully implicit Runge-Kutta (IRK) methods have many desirable properties as time integration schemes in terms of accuracy and stability, but high-order IRK methods are not commonly used in practice with numerical PDEs due to the difficulty of solving the stage equations. This paper introduces a theoretical and algorithmic preconditioning framework for solving the systems of equations that arise from IRK methods applied to linear numerical PDEs (without algebraic constraints). This framework also naturally applies to discontinuous Galerkin discretizations in time. Under quite general assumptions…
In this paper we analyze the error as well for the semi-discretization as the full discretization of a time-dependent convection-diffusion problem. We use for the discretization in space the local discontinuous Galerkin (LDG) method on a class of layer-adapted meshes including Shishkin-type and Bakhvalov-type meshes and the implicit $θ$-scheme in time. For piecewise tensor-product polynomials of degree $k$ we obtain uniform or almost uniform error estimates with respect to space of order $k+1/2$ in some energy norm and optimal error estimates with respect to time. Our analysis is based on care…
In this paper, we propose the idea of radial scaling in frequency domain and activation functions with compact support to produce a multi-scale DNN (MscaleDNN), which will have the multi-scale capability in approximating high frequency and high dimensional functions and speeding up the solution of high dimensional PDEs. Numerical results on high dimensional function fitting and solutions of high dimensional PDEs, using loss functions with either Ritz energy or least squared PDE residuals, have validated the increased power of multi-scale resolution and high frequency capturing of the proposed …
Finite element simulations have been used to solve various partial differential equations (PDEs) that model physical, chemical, and biological phenomena. The resulting discretized solutions to PDEs often do not satisfy requisite physical properties, such as positivity or monotonicity. Such invalid solutions pose both modeling challenges, since the physical interpretation of simulation results is not possible, and computational challenges, since such properties may be required to advance the scheme. We, therefore, consider the problem of computing solutions that preserve these structural soluti…
There has been an arising trend of adopting deep learning methods to study partial differential equations (PDEs). This article is to propose a Deep Learning Galerkin Method (DGM) for the closed-loop geothermal system, which is a new coupled multi-physics PDEs and mainly consists of a framework of underground heat exchange pipelines to extract the geothermal heat from the geothermal reservoir. This method is a natural combination of Galerkin Method and machine learning with the solution approximated by a neural network instead of a linear combination of basis functions. We train the neural netw…
We analyze Galerkin discretizations of a new well-posed mixed space-time variational formulation of parabolic PDEs. For suitable pairs of finite element trial spaces, the resulting Galerkin operators are shown to be uniformly stable. The method is compared to two related space-time discretization methods introduced in [IMA J. Numer. Anal., 33(1) (2013), pp. 242-260] by R. Andreev and in [Comput. Methods Appl. Math., 15(4) (2015), pp. 551-566] by O. Steinbach.
Fully implicit Runge-Kutta (IRK) methods have many desirable accuracy and stability properties as time integration schemes, but high-order IRK methods are not commonly used in practice with large-scale numerical PDEs because of the difficulty of solving the stage equations. This paper introduces a theoretical and algorithmic framework for solving the nonlinear equations that arise from IRK methods (and discontinuous Galerkin discretizations in time) applied to nonlinear numerical PDEs, including PDEs with algebraic constraints. Several new linearizations of the nonlinear IRK equations are deve…
Fourier continuation is an approach used to create periodic extensions of non-periodic functions in order to obtain highly-accurate Fourier expansions. These methods have been used in PDE-solvers and have demonstrated high-order convergence and spectrally accurate dispersion relations in numerical experiments. Discontinuous Galerkin (DG) methods are increasingly used for solving PDEs and, as all Galerkin formulations, come with a strong framework for proving stability and convergence. Here we propose the use of Fourier continuation in forming a new basis for the DG framework.
