Neural Network Differential Equation (NN DE) solvers have surged in popularity due to a combination of factors: computational advances making their optimization more tractable, their capacity to handle high dimensional problems, easy interpret-ability of their models, etc. However, almost all NN DE solvers suffer from a fundamental limitation: they are trained using loss functions that depend only implicitly on the error associated with the estimate. As such, validation and error analysis of solution estimates requires knowledge of the true solution. Indeed, if the true solution is unknown, we are often reduced to simply hoping that a "low enough" loss implies "small enough" errors, since explicit relationships between the two are not available/well defined. In this work, we describe a general strategy for efficiently constructing error estimates and corrections for Neural Network Differential Equation solvers. Our methods do not require advance knowledge of the true solutions and obtain explicit relationships between loss functions and the error associated with solution estimates. In turn, these explicit relationships directly allow us to estimate and correct for the errors.
Traditional solvers for delay differential equations (DDEs) are designed around only a single method and do not effectively use the infrastructure of their more-developed ordinary differential equation (ODE) counterparts. In this work we present DelayDiffEq, a Julia package for numerically solving delay differential equations (DDEs) which leverages the multitude of numerical algorithms in OrdinaryDiffEq for solving both stiff and non-stiff ODEs, and manages to solve challenging stiff DDEs. We describe how compiling the ODE integrator within itself, and accounting for discontinuity propagation,…
Technical University Munich（慕尼黑工业大学）；École normale supérieure, Paris（巴黎高等师范学院）；deepc GmbH, Munich（慕尼黑deepc有限公司）；Helmholtz Zentrum München（慕尼黑亥姆霍兹研究中心）；University of Zurich（苏黎世大学）
Fast and accurate solutions of time-dependent partial differential equations (PDEs) are of pivotal interest to many research fields, including physics, engineering, and biology. Generally, implicit/semi-implicit schemes are preferred over explicit ones to improve stability and correctness. However, existing semi-implicit methods are usually iterative and employ a general-purpose solver, which may be sub-optimal for a specific class of PDEs. In this paper, we propose a neural solver to learn an optimal iterative scheme in a data-driven fashion for any class of PDEs. Specifically, we modify a si…
We motivate the use of neural networks for the construction of numerical solutions to differential equations. We prove that there exists a feed-forward neural network that can arbitrarily minimise an objective function that is zero at the solution of Poisson's equation, allowing us to guarantee that neural network solution estimates can get arbitrarily close to the exact solutions. We also show how these estimates can be appreciably enhanced through various strategies, in particular through the construction of error correction networks, for which we propose a general method. We conclude by pro…
Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Julia Computing；Massachusetts Institute of Technology（麻省理工学院）；University of Maryland Baltimore（马里兰大学巴尔的摩分校）；Pumas AI
Democratization of machine learning requires architectures that automatically adapt to new problems. Neural Differential Equations (NDEs) have emerged as a popular modeling framework by removing the need for ML practitioners to choose the number of layers in a recurrent model. While we can control the computational cost by choosing the number of layers in standard architectures, in NDEs the number of neural network evaluations for a forward pass can depend on the number of steps of the adaptive ODE solver. But, can we force the NDE to learn the version with the least steps while not increasing…
The numerical solution methods for partial differential equation (PDE) solution allow obtaining a discrete field that converges towards the solution if the method is applied to the correct problem. Nevertheless, the numerical methods usually have the restricted class of the equations, on which the convergence is proved. Only a small amount of "cheap and dirty" numerical methods converge on a wide class of equations with the lower approximation order price. In the article, we present a method that uses an optimization algorithm to obtain a solution that could be used as the initial guess for th…
On the computation of accurate initial conditions for linear higher-index differential-algebraic equations and its application in initial value solvers
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Probabilistic numerical solvers for ordinary differential equations compute posterior distributions over the solution of an initial value problem via Bayesian inference. In this paper, we leverage their probabilistic formulation to seamlessly include additional information as general likelihood terms. We show that second-order differential equations should be directly provided to the solver, instead of transforming the problem to first order. Additionally, by including higher-order information or physical conservation laws in the model, solutions become more accurate and more physically meanin…
Machine learned partial differential equation (PDE) solvers trade the reliability of standard numerical methods for potential gains in accuracy and/or speed. The only way for a solver to guarantee that it outputs the exact solution is to use a convergent method in the limit that the grid spacing $Δx$ and timestep $Δt$ approach zero. Machine learned solvers, which learn to update the solution at large $Δx$ and/or $Δt$, can never guarantee perfect accuracy. Some amount of error is inevitable, so the question becomes: how do we constrain machine learned solvers to give us the sorts of errors that…
We present a highly scalable strategy for developing mesh-free neuro-symbolic partial differential equation solvers from existing numerical discretizations found in scientific computing. This strategy is unique in that it can be used to efficiently train neural network surrogate models for the solution functions and the differential operators, while retaining the accuracy and convergence properties of state-of-the-art numerical solvers. This neural bootstrapping method is based on minimizing residuals of discretized differential systems on a set of random collocation points with respect to the…
We present a scalable and efficient iterative solver for high-order hybridized discontinuous Galerkin (HDG) discretizations of hyperbolic partial differential equations. It is an interplay between domain decomposition methods and HDG discretizations. In particular, the method is a fixed-point approach that requires only independent element-by-element local solves in each iteration. As such, it is well-suited for current and future computing systems with massive concurrencies. We rigorously show that the proposed method is exponentially convergent in the number of iterations for transport and l…
A new method for solving non-autonomous ordinary differential equations is proposed, the method achieves spectral accuracy. It is based on a new result which expresses the solution of such ODEs as an element in the so called $\star$-algebra. This algebra is equipped with a product, the $\star$-product, which is the integral over the usual product of two bivariate distributions. Expanding the bivariate distributions in bases of Legendre polynomials leads to a discretization of the $\star$-product and this allows for the solution to be approximated by a vector that is obtained by solving a linea…
A Structure-Preserving Framework for Solving Parabolic Partial Differential Equations with Neural Networks
Department of Applied Mathematics, The Hong Kong Polytechnic University（应用数学系，香港Polytechnic大学）；Department of Mathematics, University of South Carolina（数学系，南卡罗来纳大学）
University of Nevada, Las Vegas（内华达大学拉斯维加斯分校）；University of Arkansas（阿肯色大学）；Lehigh University（里海大学）
In this paper, we show a physics-informed neural network solver for the time-dependent surface PDEs. Unlike the traditional numerical solver, no extension of PDE and mesh on the surface is needed. We show a simplified prior estimate of the surface differential operators so that PINN's loss value will be an indicator of the residue of the surface PDEs. Numerical experiments verify efficacy of our algorithm.
An efficient solver based on low-rank approximation and Neumann matrix series for unsteady diffusion-type partial differential equations with random coefficients
School of Mathematics and Statistics, Huazhong University of Science and Technology（华中科技大学数学与统计学院）；Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology（华中科技大学工程建模与科学计算湖北省重点实验室）
We present an arbitrarily high-order, conditionally stable, partitioned spectral deferred correction (SDC) method for solving multiphysics problems using a sequence of pre-existing single-physics solvers. This method extends the work in [1, 2], which used implicit-explicit Runge-Kutta methods (IMEX) to build high-order, partitioned multiphysics solvers. We consider a generic multiphysics problem modeled as a system of coupled ordinary differential equations (ODEs), coupled through coupling terms that can depend on the state of each subsystem; therefore the method applies to both a semi-discret…
We present a Fourier neural network (FNN) that can be mapped directly to the Fourier decomposition. The choice of activation and loss function yields results that replicate a Fourier series expansion closely while preserving a straightforward architecture with a single hidden layer. The simplicity of this network architecture facilitates the integration with any other higher-complexity networks, at a data pre- or postprocessing stage. We validate this FNN on naturally periodic smooth functions and on piecewise continuous periodic functions. We showcase the use of this FNN for modeling or solvi…
Scalable Multigrid Solver for the Helmholtz Equation: Real-Shifted Coarse Grid Correction
Institute for Interdisciplinary Computational Science, the Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev（内盖夫本-古里安大学斯坦计算机与信息科学学院跨学科计算科学研究所）
Traditional, numerical discretization-based solvers of partial differential equations (PDEs) are fundamentally agnostic to domains, boundary conditions and coefficients. In contrast, machine learnt solvers have a limited generalizability across these elements of boundary value problems. This is strongly true in the case of surrogate models that are typically trained on direct numerical simulations of PDEs applied to one specific boundary value problem. In a departure from this direct approach, the label-free machine learning of solvers is centered on a loss function that incorporates the PDE a…
This article presents several numerical techniques for solving Laplace equation. A numerical FORTRAN solver is developed to solve the 2D laplace equation. The numerical approaches implemented in the solver include Jacobi, Gauss-Siedel, Successive Over Relaxation, Alternating Direct Implicit and Multigrid methods. Detailed comparison between different numerical methods is presented and discussed
The discovery of fast numerical solvers prompted a clear and rapid shift towards iterative techniques in many applications, especially in computational mechanics, due to the increased necessity for solving very large linear systems. Most numerical solvers are highly dependent on the problem geometry and discretization, facing issues when any of these properties change. The newly developed Hybrid Iterative Numerical Transferable Solver (HINTS) combines a standard solver with a neural operator to achieve better performance, focusing on a single geometry at a time. In this work, we explore the "T…
Stochastic differential equations play an important role in various applications when modeling systems that have either random perturbations or chaotic dynamics at faster time scales. The time evolution of the probability distribution of a stochastic differential equation is described by the Fokker-Planck equation, which is a second order parabolic partial differential equation. Previous work combined artificial neural network and Monte Carlo data to solve stationary Fokker-Planck equations. This paper extends this approach to time dependent Fokker-Planck equations. The focus is on the investi…
Deep Parallel Spectral Neural Operators for Solving Partial Differential Equations with Enhanced Low-Frequency Learning Capability
College of Computer Science and Technology（计算机科学与技术学院）；China University of Petroleum (East China)（中国石油大学（华东））；State Key Laboratory of Chemical Safety（化学安全国家重点实验室）
The time-marching strategy, which propagates the solution from one time step to the next, is a natural strategy for solving time-dependent differential equations on classical computers, as well as for solving the Hamiltonian simulation problem on quantum computers. For more general linear differential equations, a time-marching based quantum solver can suffer from exponentially vanishing success probability with respect to the number of time steps and is thus considered impractical. We solve this problem by repeatedly invoking a technique called the uniform singular value amplification, and th…
This work develops a class of probabilistic algorithms for the numerical solution of nonlinear, time-dependent partial differential equations (PDEs). Current state-of-the-art PDE solvers treat the space- and time-dimensions separately, serially, and with black-box algorithms, which obscures the interactions between spatial and temporal approximation errors and misguides the quantification of the overall error. To fix this issue, we introduce a probabilistic version of a technique called method of lines. The proposed algorithm begins with a Gaussian process interpretation of finite difference m…
In this paper we consider utilizing a residual neural network (ResNet) to solve ordinary differential equations. Stochastic gradient descent method is applied to obtain the optimal parameter set of weights and biases of the network. We apply forward Euler, Runge-Kutta2 and Runge-Kutta4 finite difference methods to generate three sets of targets training the ResNet and carry out the target study. The well trained ResNet behaves just as its counterpart of the corresponding one-step finite difference method. In particular, we carry out (1) the architecture study in terms of number of hidden layer…
Multi-patch isogeometric neural solver for partial differential equations on computer-aided design domains
Institute for Accelerator Science and Electromagnetic Fields, Technische Universität Darmstadt（加速器科学与电磁场研究所，德累斯顿技术大学）；Terra Quantum AG（Terra Quantum公司）；Scientific Computing, Centrum Wiskunde & Informatica（科学计算，数学与信息学中心）
University of Strathclyde（思克莱德大学）；Nanjing University of Aeronautics & Astronautics（南京航空航天大学）
This paper presents a novel and fast solver for the J2-perturbed Lambert problem. The solver consists of an intelligent initial guess generator combined with a differential correction procedure. The intelligent initial guess generator is a deep neural network that is trained to correct the initial velocity vector coming from the solution of the unperturbed Lambert problem. The differential correction module takes the initial guess and uses a forward shooting procedure to further update the initial velocity and exactly meet the terminal conditions. Eight sample forms are analyzed and compared t…
