We propose quadratic residual networks (QRes) as a new type of parameter-efficient neural network architecture, by adding a quadratic residual term to the weighted sum of inputs before applying activation functions. With sufficiently high functional capacity (or expressive power), we show that it is especially powerful for solving forward and inverse physics problems involving partial differential equations (PDEs). Using tools from algebraic geometry, we theoretically demonstrate that, in contrast to plain neural networks, QRes shows better parameter efficiency in terms of network width and depth thanks to higher non-linearity in every neuron. Finally, we empirically show that QRes shows faster convergence speed in terms of number of training epochs especially in learning complex patterns.
In recent years residual neural networks (ResNets) as introduced by [He, K., Zhang, X., Ren, S., and Sun, J., Proceedings of the IEEE conference on computer vision and pattern recognition (2016), 770-778] have become very popular in a large number of applications, including in image classification and segmentation. They provide a new perspective in training very deep neural networks without suffering the vanishing gradient problem. In this article we show that ResNets are able to approximate solutions of Kolmogorov partial differential equations (PDEs) with constant diffusion and possibly nonl…
We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neura…
In this paper, we propose the idea of radial scaling in frequency domain and activation functions with compact support to produce a multi-scale DNN (MscaleDNN), which will have the multi-scale capability in approximating high frequency and high dimensional functions and speeding up the solution of high dimensional PDEs. Numerical results on high dimensional function fitting and solutions of high dimensional PDEs, using loss functions with either Ritz energy or least squared PDE residuals, have validated the increased power of multi-scale resolution and high frequency capturing of the proposed …
SPUS: A Lightweight and Parameter-Efficient Foundation Model for PDEs
Computing and Artificial Intelligence Division (CAI)（计算与人工智能部门）；Space Remote Sensing and Data Science（空间遥感与数据科学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
PG-KINN: A Physics-Informed Petrov-Galerkin Kolmogorov-Arnold Network for Solving Forward and Inverse PDEs
Shahid Beheshti University（沙希德·贝赫什提大学）；Institute for Research in Fundamental Sciences (IPM)（基础科学研究所（IPM））；Cyberspace Research Institute, Shahid Beheshti University（沙希德·贝赫什提大学网络空间研究所）；University of Hertfordshire（赫特福德大学）
LLM-PDESR: Robust PDE Discovery via Subdomain Weighted Residuals and LLM-Guided Symbolic Hypothesis Generation
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；School of Big Data and Artificial Intelligence, Guangdong University of Finance and Economics（广东财经大学大数据与人工智能学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（吉林大学符号计算与知识工程教育部重点实验室）；School of Computer Science, Zhuhai College of Science and Technology（珠海科技学院计算机科学学院）；Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）
Adaptive Randomized Neural Networks for PDEs: Structure-Aware Feature Redistribution and Residual-Driven Block Enrichment
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；School of Mathematics and Statistics & State Key Laboratory of Multiphase Flow in Power Engineering, Xi’an Jiaotong University（西安交通大学数学与统计学院）
Defense Innovation Institute, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）；Human Resource Center（人力资源中心）
Learning solutions of partial differential equations (PDEs) with Physics-Informed Neural Networks (PINNs) is an attractive alternative approach to traditional solvers due to its flexibility and ease of incorporating observed data. Despite the success of PINNs in accurately solving a wide variety of PDEs, the method still requires improvements in terms of computational efficiency. One possible improvement idea is to optimize the generation of training point sets. Residual-based adaptive sampling and quasi-uniform sampling approaches have been each applied to improve the training effects of PINN…
RAMS: Residual-based adversarial-gradient moving sample method for scientific machine learning in solving partial differential equations
Department of Civil and Environmental Engineering, Hong Kong Polytechnic University（香港理工大学土木与环境工程系）；Department of Statistics and Data Science, Yale University（耶鲁大学统计与数据科学系）
Learning-guided Kansa collocation for forward and inverse PDEs beyond linearity
Department of Computer Science and Technology（计算机科学与技术系）；Department of Applied Mathematics and Theoretical Physics（应用数学与理论物理系）；University of Cambridge（剑桥大学）
Physics informed neural networks approximate solutions of PDEs by minimizing pointwise residuals. We derive rigorous bounds on the error, incurred by PINNs in approximating the solutions of a large class of linear parabolic PDEs, namely Kolmogorov equations that include the heat equation and Black-Scholes equation of option pricing, as examples. We construct neural networks, whose PINN residual (generalization error) can be made as small as desired. We also prove that the total $L^2$-error can be bounded by the generalization error, which in turn is bounded in terms of the training error, prov…
In this paper, we perform the convergence analysis of unsupervised Legendre--Galerkin neural networks (ULGNet), a deep-learning-based numerical method for solving partial differential equations (PDEs). Unlike existing deep learning-based numerical methods for PDEs, the ULGNet expresses the solution as a spectral expansion with respect to the Legendre basis and predicts the coefficients with deep neural networks by solving a variational residual minimization problem. Since the corresponding loss function is equivalent to the residual induced by the linear algebraic system depending on the choic…
Cornell University（康奈尔大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Hawai’i at Manoa（夏威夷大学马诺阿分校）
This work is the first to employ and adapt the image-to-image translation concept based on conditional generative adversarial networks (cGAN) towards learning a forward and an inverse solution operator of partial differential equations (PDEs). Even though the proposed framework could be applied as a surrogate model for the solution of any PDEs, here we focus on steady-state solutions of coupled hydro-mechanical processes in heterogeneous porous media. Strongly heterogeneous material properties, which translate to the heterogeneity of coefficients of the PDEs and discontinuous features in the s…
Solving partial differential equations (PDEs) is the canonical approach for understanding the behavior of physical systems. However, large scale solutions of PDEs using state of the art discretization techniques remains an expensive proposition. In this work, a new physics-constrained neural network (NN) approach is proposed to solve PDEs without labels, with a view to enabling high-throughput solutions in support of design and decision-making. Distinct from existing physics-informed NN approaches, where the strong form or weak form of PDEs are used to construct the loss function, we write the…
Adversarial Adaptive Sampling: Unify PINN and Optimal Transport for the Approximation of PDEs
Louisiana State University（路易斯安那州立大学）；Peking University（北京大学）；PKU-Changsha Institute for Computing and Digital Economy（北京大学-长沙计算与数字经济研究院）；ShanghaiTech University（上海科技大学）
Applications in quantitative finance such as optimal trade execution, risk management of options, and optimal asset allocation involve the solution of high dimensional and nonlinear Partial Differential Equations (PDEs). The connection between PDEs and systems of Forward-Backward Stochastic Differential Equations (FBSDEs) enables the use of advanced simulation techniques to be applied even in the high dimensional setting. Unfortunately, when the underlying application contains nonlinear terms, then classical methods both for simulation and numerical methods for PDEs suffer from the curse of di…
A Physics-Informed Meta-Learning Framework for the Continuous Solution of Parametric PDEs on Arbitrary Geometries
Graduate School of Science and Technology, Keio University（Keio大学科学技术研究生院）；Department of Mechanical Engineering, Keio University（Keio大学机械工程系）；Access e.V.
Minzu University of China（中央民族大学）；North China University of Technology（北方工业大学）
As a typical application of deep learning, physics-informed neural network (PINN) {has been} successfully used to find numerical solutions of partial differential equations (PDEs), but how to improve the limited accuracy is still a great challenge for PINN. In this work, we introduce a new method, symmetry-enhanced physics informed neural network (SPINN) where the invariant surface conditions induced by the Lie symmetries or non-classical symmetries of PDEs are embedded into the loss function in PINN, to improve the accuracy of PINN for solving the forward and inverse problems of PDEs. We test…
Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov Arnold Networks
Tsinghua University（清华大学）；Bauhaus-Universität Weimar（魏玛包豪斯大学）；CNPC Engineering Technology RD Company Limited（中国石油集团工程技术研究院有限公司）；Queensland University of Technology（昆士兰科技大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
Northeastern University（东北大学）；University of Notre Dame（圣母大学）；MIT（麻省理工学院）
Partial differential equations (PDEs) play a fundamental role in modeling and simulating problems across a wide range of disciplines. Recent advances in deep learning have shown the great potential of physics-informed neural networks (PINNs) to solve PDEs as a basis for data-driven modeling and inverse analysis. However, the majority of existing PINN methods, based on fully-connected NNs, pose intrinsic limitations to low-dimensional spatiotemporal parameterizations. Moreover, since the initial/boundary conditions (I/BCs) are softly imposed via penalty, the solution quality heavily relies on h…
Gated X-TFC: Soft Domain Decomposition for Forward and Inverse Problems in Sharp-Gradient PDEs
CREATIS Biomedical Imaging Laboratory INSA, CNRS UMR 5220, Inserm, Universit´e Lyon 1（CREATIS生物医学成像实验室，INSA，CNRS UMR 5220，Inserm，里昂1大学）；Department of Industrial Engineering University of Bologna（工业工程系，博洛尼亚大学）
Multi-Fidelity Physics-Informed Neural Networks with Bayesian Uncertainty Quantification and Adaptive Residual Learning for Efficient Solution of Parametric Partial Differential Equations
Department of Applied Mathematics and Computer Science (DTU Compute)（应用数学与计算机科学系（DTU Compute））；Technical University of Denmark（丹麦技术大学）
St. Mark’s School of Texas（圣马克德克萨斯学校）；University of Pennsylvania（宾夕法尼亚大学）；Brown University（布朗大学）
Deep learning has been shown to be an effective tool in solving partial differential equations (PDEs) through physics-informed neural networks (PINNs). PINNs embed the PDE residual into the loss function of the neural network, and have been successfully employed to solve diverse forward and inverse PDE problems. However, one disadvantage of the first generation of PINNs is that they usually have limited accuracy even with many training points. Here, we propose a new method, gradient-enhanced physics-informed neural networks (gPINNs), for improving the accuracy and training efficiency of PINNs.…
Training deep recurrent neural network (RNN) architectures is complicated due to the increased network complexity. This disrupts the learning of higher order abstracts using deep RNN. In case of feed-forward networks training deep structures is simple and faster while learning long-term temporal information is not possible. In this paper we propose a residual memory neural network (RMN) architecture to model short-time dependencies using deep feed-forward layers having residual and time delayed connections. The residual connection paves way to construct deeper networks by enabling unhindered f…
Shanghai Normal University（上海师范大学）；Tongji University（同济大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
We introduce a sampling based machine learning approach, Monte Carlo physics informed neural networks (MC-PINNs), for solving forward and inverse fractional partial differential equations (FPDEs). As a generalization of physics informed neural networks (PINNs), our method relies on deep neural network surrogates in addition to a stochastic approximation strategy for computing the fractional derivatives of the DNN outputs. A key ingredient in our MC-PINNs is to construct an unbiased estimation of the physical soft constraints in the loss function. Our directly sampling approach can yield less o…
Neural Measures for learning distributions of Random PDEs
DMAM, University of Crete, Greece（DMAM，希腊克里特大学）；IACM, Foundation for Research and Technology - Hellas, Greece（IACM，希腊基础研究与技术研究所）；MPS, University of Sussex, United Kingdom（MPS，英国 Sussex 大学）
UT-Battelle, LLC（UT-巴特尔有限责任公司）；US Department of Energy (DOE)（美国能源部）
In this effort, we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show that this choice leads to the improved stability of both forward and backward propagations, has a favorable impact on the generalization power and allows to control the robustness of the network with only a few hyperparameters. In addition, the proposed reformulation of ResNet do…
Physics-informed neural networks (PINNs) have been proposed to learn the solution of partial differential equations (PDE). In PINNs, the residual form of the PDE of interest and its boundary conditions are lumped into a composite objective function as soft penalties. Here, we show that this specific way of formulating the objective function is the source of severe limitations in the PINN approach when applied to different kinds of PDEs. To address these limitations, we propose a versatile framework based on a constrained optimization problem formulation, where we use the augmented Lagrangian m…
We propose characteristics-informed neural networks (CINN), a simple and efficient machine learning approach for solving forward and inverse problems involving hyperbolic PDEs. Like physics-informed neural networks (PINN), CINN is a meshless machine learning solver with universal approximation capabilities. Unlike PINN, which enforces a PDE softly via a multi-part loss function, CINN encodes the characteristics of the PDE in a general-purpose deep neural network by adding a characteristic layer. This neural network is trained with the usual MSE data-fitting regression loss and does not require…
Physics-informed neural networks (PINNs) as a means of discretizing partial differential equations (PDEs) are garnering much attention in the Computational Science and Engineering (CS&E) world. At least two challenges exist for PINNs at present: an understanding of accuracy and convergence characteristics with respect to tunable parameters and identification of optimization strategies that make PINNs as efficient as other computational science tools. The cost of PINNs training remains a major challenge of Physics-informed Machine Learning (PiML) - and, in fact, machine learning (ML) in general…
Extended Dynamic Mode Decomposition (EDMD) is a popular data-driven method to approximate the action of the Koopman operator on a linear function space spanned by a dictionary of functions. The accuracy of EDMD model critically depends on the quality of the particular dictionary's span, specifically on how close it is to being invariant under the Koopman operator. Motivated by the observation that the residual error of EDMD, typically used for dictionary learning, does not encode the quality of the function space and is sensitive to the choice of basis, we introduce the novel concept of consis…
This work presents a non-intrusive surrogate modeling scheme based on machine learning technology for predictive modeling of complex systems, described by parametrized time-dependent PDEs. For these problems, typical finite element approaches involve the spatiotemporal discretization of the PDE and the solution of the corresponding linear system of equations at each time step. Instead, the proposed method utilizes a convolutional autoencoder in conjunction with a feed forward neural network to establish a low-cost and accurate mapping from the problem's parametric space to its solution space. …
Universität Potsdam（波茨坦大学）；Freie Universität Berlin（柏林自由大学）；Brandenburgische Technische Universität Cottbus-Senftenberg（勃兰登堡工业大学科特布斯-森夫滕贝格）
Optimal control of diffusion processes is intimately connected to the problem of solving certain Hamilton-Jacobi-Bellman equations. Building on recent machine learning inspired approaches towards high-dimensional PDEs, we investigate the potential of $\textit{iterative diffusion optimisation}$ techniques, in particular considering applications in importance sampling and rare event simulation, and focusing on problems without diffusion control, with linearly controlled drift and running costs that depend quadratically on the control. More generally, our methods apply to nonlinear parabolic PDEs…
