Translating Numerical Concepts for PDEs into Neural Architectures
Saarland University（萨尔兰大学）
Tobias Alt, Pascal Peter, Joachim Weickert, Karl Schrader
备注：
In A. Elmoataz, J. Fadili, Y. Quéau, J. Rabin, L. Simon (Eds.): Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 12679, Springer, Cham, 294-306, 2021
We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent networks, and U-nets. These connections guarantee Euclidean stability of specific ResNets with a transposed convolution layer structure in each block. We present three numerical justifications for skip connections: as time discretisations in explicit schemes, as extrapolation mechanisms for accelerating those methods, and as recurrent connections in fixed point solvers for implicit schemes. Last but not least, we also motivate uncommon design choices such as nonmonotone activation functions. Our findings give a numerical perspective on the success of modern neural network architectures, and they provide design criteria for stable networks.
We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neura…
Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in a corresponding rotation of the output, which is desirable in applications such as image analysis. Convolutional neural networks (CNNs) do not share this property, and existing remedies are often complex. The goal of our paper is to investigate how diffusion and variational models achieve rotation invariance and transfer these ideas to neural networks. As a core novelty we propose activation functions wh…
Cornell University（康奈尔大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Hawai’i at Manoa（夏威夷大学马诺阿分校）
This work is the first to employ and adapt the image-to-image translation concept based on conditional generative adversarial networks (cGAN) towards learning a forward and an inverse solution operator of partial differential equations (PDEs). Even though the proposed framework could be applied as a surrogate model for the solution of any PDEs, here we focus on steady-state solutions of coupled hydro-mechanical processes in heterogeneous porous media. Strongly heterogeneous material properties, which translate to the heterogeneity of coefficients of the PDEs and discontinuous features in the s…
This paper develops a unified general framework for designing convergent finite difference and discontinuous Galerkin methods for approximating viscosity and regular solutions of fully nonlinear second order PDEs. Unlike the well-known monotone (finite difference) framework, the proposed new framework allows for the use of narrow stencils and unstructured grids which makes it possible to construct high order methods. The general framework is based on the concepts of consistency and g-monotonicity which are both defined in terms of various numerical derivative operators. Specific methods that s…
A novel multi-level method for partial differential equations with uncertain parameters is proposed. The principle behind the method is that the error between grid levels in multi-level methods has a spatial structure that is by good approximation independent of the actual grid level. Our method learns this structure by employing a sequence of convolutional neural networks, that are well-suited to automatically detect local error features as latent quantities of the solution. Furthermore, by using the concept of transfer learning, the information of coarse grid levels is reused on fine grid le…
As further progress in the accurate and efficient computation of coupled partial differential equations (PDEs) becomes increasingly difficult, it has become highly desired to develop new methods for such computation. In deviation from conventional approaches, this short communication paper explores a computational paradigm that couples numerical solutions of PDEs via machine-learning (ML) based methods, together with a preliminary study on the paradigm. Particularly, it solves PDEs in subdomains as in a conventional approach but develops and trains artificial neural networks (ANN) to couple th…
Recently, neural networks have been widely applied for solving partial differential equations (PDEs). Although such methods have been proven remarkably successful on practical engineering problems, they have not been shown, theoretically or empirically, to converge to the underlying PDE solution with arbitrarily high accuracy. The primary difficulty lies in solving the highly non-convex optimization problems resulting from the neural network discretization, which are difficult to treat both theoretically and practically. It is our goal in this work to take a step toward remedying this. For thi…
Isogeometric Analysis for Surface PDEs with Extended Loop Subdivision
Hunan Normal University（湖南师范大学）；Hangzhou Dianzi University（杭州电子科技大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
The Active Flux scheme is a Finite Volume scheme with additional point values distributed along the cell boundary. It is third order accurate and does not require a Riemann solver: the continuous reconstruction serves as initial data for the evolution of the points values. The intercell flux is then obtained from the evolved values along the cell boundary by quadrature. This paper focuses on the conceptual extension of Active Flux to include source terms, and thus for simplicity assumes the homogeneous part of the equations to be linear. To a large part, the treatment of the source terms is in…
Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems
Centre for Industrial Mathematics (ZeTeM), University of Bremen（不来梅大学工业数学中心（ZeTeM））；School of Mathematics and Statistics, Wuhan University（武汉大学数学与统计学院）；Robert Bosch GmbH（罗伯特·博世有限公司）
Comparison of neural closure models for discretised PDEs
Eindhoven University of Technology（埃因霍温理工大学）；Centrum Wiskunde & Informatica（荷兰数学与计算机科学研究院）；Korteweg-de Vries Institute for Mathematics, University of Amsterdam（阿姆斯特丹大学科特韦格-德弗里斯数学研究所）
In the present work, a multi-scale framework for neural network enhanced methods is proposed for approximation of function and solution of partial differential equations (PDEs). By introducing the multi-scale concept, the total solution of the target problem could be decomposed into two parts, i.e. the coarse scale solution and the fine scale solution. In the coarse scale, the conventional numerical methods (e.g. finite element methods) are applied and the coarse scale solution could be obtained. In the fine scale, the neural networks is introduced to formulate the solution. The custom loss fu…
Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations
Department One（第一部门）；Department Two（第二部门）；Department of Mathematics（数学系）；The Hong Kong University of Science and Technology（香港科技大学）；The Pennsylvania State University（宾夕法尼亚州立大学）；Algorithms of Machine Learning and Autonomous Driving Research Lab（机器学习与自动驾驶算法研究实验室）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
本文研究神经网络求解 PDE 时 AD 相对 FD 的训练优势，提出截断熵指标，并通过随机特征模型与两层网络的实验和理论分析证明 AD 训练更快、更可靠。
Deep Neural networks for solving high-dimensional parabolic partial differential equations
Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州研究院）；Department of Mathematics, Southern Methodist University（南方法国大学数学系）；Department of Computer Science, School of Computing, National University of Singapore（新加坡国立大学计算机科学系）；Department of Mathematics, Southern Methodist University (SMU)（南方法国大学数学系）；Division of Applied Mathematics, Brown University（布朗大学应用数学系）
Parametrized families of PDEs arise in various contexts such as inverse problems, control and optimization, risk assessment, and uncertainty quantification. In most of these applications, the number of parameters is large or perhaps even infinite. Thus, the development of numerical methods for these parametric problems is faced with the possible curse of dimensionality. This article is directed at (i) identifying and understanding which properties of parametric equations allow one to avoid this curse and (ii) developing and analyzing effective numerical methodd which fully exploit these proper…
The numerical solution of differential equations can be formulated as an inference problem to which formal statistical approaches can be applied. However, nonlinear partial differential equations (PDEs) pose substantial challenges from an inferential perspective, most notably the absence of explicit conditioning formula. This paper extends earlier work on linear PDEs to a general class of initial value problems specified by nonlinear PDEs, motivated by problems for which evaluations of the right-hand-side, initial conditions, or boundary conditions of the PDE have a high computational cost. Th…
School of Mathematical Sciences（数学科学学院）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Peking University, Beijing, China（北京大学北京中国）；Beijing Computational Science Research Center（北京计算科学研究中心）；Beijing International Center for Mathematical Research, Peking University（北京大学北京国际数学研究中心）；Center for Data Science, Peking University（北京大学数据科学中心）；Beijing Institute of Big Data Research（北京大数据研究院）
Infinite-dimensional Newton methods can be effectively used to derive numerical proofs of the existence of solutions to partial differential equations (PDEs). In computer-assisted proofs of PDEs, the original problem is transformed into the infinite Newton-type fixed point equation $w = - {\mathcal L}^{-1} {\mathcal F}(\hat{u}) + {\mathcal L}^{-1} {\mathcal G}(w)$, where ${\mathcal L}$ is a linearized operator, ${\mathcal F}(\hat{u})$ is a residual, and ${\mathcal G}(w)$ is a local Lipschitz term. Therefore, the estimations of $\| {\mathcal L}^{-1} {\mathcal F}(\hat{u}) \|$ and $\| {\mathcal L…
