Recently, there has been a lot of interest in using neural networks for solving partial differential equations. A number of neural network-based partial differential equation solvers have been formulated which provide performances equivalent, and in some cases even superior, to classical solvers. However, these neural solvers, in general, need to be retrained each time the initial conditions or the domain of the partial differential equation changes. In this work, we posit the problem of approximating the solution of a fixed partial differential equation for any arbitrary initial conditions as learning a conditional probability distribution. We demonstrate the utility of our method on Burger's Equation.
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Google Brain（谷歌大脑）
Differential equations parameterized by neural networks become expensive to solve numerically as training progresses. We propose a remedy that encourages learned dynamics to be easier to solve. Specifically, we introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. These derivatives are efficient to compute with Taylor-mode automatic differentiation. Optimizing this additional objective trades model performance against the time cost of solving the learned dynamics. We demonstrate our approach by training su…
This paper investigates the use of artificial neural networks (ANNs) to solve differential equations (DEs) and the construction of the loss function which meets both differential equation and its initial/boundary condition of a certain DE. In section 2, the loss function is generalized to $n^\text{th}$ order ordinary differential equation(ODE). Other methods of construction are examined in Section 3 and applied to three different models to assess their effectiveness.
University of Ontario Institute of Technology（安大略理工大学）
We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden layer is general, and that deeper layers are successively more specific. Next, we validate our method against an existing technique that measures layer generality using transfer learning experiments. …
NewPINNs: Physics-Informing Neural Networks Using Conventional Solvers for Partial Differential Equations
Department of Mathematics, University of California Riverside（加州大学河滨分校数学系）；Department of Mechanical Engineering, University of California Riverside（加州大学河滨分校机械工程系）；German Innovation Center, Procter & Gamble Service GmbH（普罗cter与 gamble 服务德国创新中心）
We present Elvet, a Python package for solving differential equations and variational problems using machine learning methods. Elvet can deal with any system of coupled ordinary or partial differential equations with arbitrary initial and boundary conditions. It can also minimize any functional that depends on a collection of functions of several variables while imposing constraints on them. The solution to any of these problems is represented as a neural network trained to produce the desired function.
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss function. However, such an approach is tailored for a specific discretization method and its properties, which may not be optimal for the selected application and yield the overfitting to the given solver. In our paper, we investigate how the variability in solvers' space can improve neural ODEs performance. We consider a family of Runge-Kutta methods that are parameterized by no more than two scalar variables. Based on the…
Helmholtz-Zentrum Dresden-Rossendorf（德累斯顿-罗森多夫亥姆霍兹中心）；Technische Universität Dresden（德累斯顿工业大学）；Center for Advanced Systems Understanding (CASUS)（高级系统理解中心）
Solving partial differential equations (PDE) is an indispensable part of many branches of science as many processes can be modelled in terms of PDEs. However, recent numerical solvers require manual discretization of the underlying equation as well as sophisticated, tailored code for distributed computing. Scanning the parameters of the underlying model significantly increases the runtime as the simulations have to be cold-started for each parameter configuration. Machine Learning based surrogate models denote promising ways for learning complex relationship among input, parameter and solution…
The neural network-based approach to solving partial differential equations has attracted considerable attention due to its simplicity and flexibility in representing the solution of the partial differential equation. In training a neural network, the network learns global features corresponding to low-frequency components while high-frequency components are approximated at a much slower rate. For a class of equations in which the solution contains a wide range of scales, the network training process can suffer from slow convergence and low accuracy due to its inability to capture the high-fre…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Indian Institute of Science Education and Research, Tirupati（印度科学教育与研究学院蒂鲁帕蒂分校）；Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
Purpose: We seek to use neural networks (NNs) to solve a well-known system of differential equations describing the balance between T cells and HIV viral burden. Materials and Methods: In this paper, we employ a 3-input parallel NN to approximate solutions for the system of first-order ordinary differential equations describing the above biochemical relationship. Results: The numerical results obtained by the NN are very similar to a host of numerical approximations from the literature. Conclusion: We have demonstrated use of NN integration of a well-known and medically important system of fir…
Neural Network Differential Equation (NN DE) solvers have surged in popularity due to a combination of factors: computational advances making their optimization more tractable, their capacity to handle high dimensional problems, easy interpret-ability of their models, etc. However, almost all NN DE solvers suffer from a fundamental limitation: they are trained using loss functions that depend only implicitly on the error associated with the estimate. As such, validation and error analysis of solution estimates requires knowledge of the true solution. Indeed, if the true solution is unknown, we…
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Probabilistic numerical solvers for ordinary differential equations compute posterior distributions over the solution of an initial value problem via Bayesian inference. In this paper, we leverage their probabilistic formulation to seamlessly include additional information as general likelihood terms. We show that second-order differential equations should be directly provided to the solver, instead of transforming the problem to first order. Additionally, by including higher-order information or physical conservation laws in the model, solutions become more accurate and more physically meanin…
We present a Fourier neural network (FNN) that can be mapped directly to the Fourier decomposition. The choice of activation and loss function yields results that replicate a Fourier series expansion closely while preserving a straightforward architecture with a single hidden layer. The simplicity of this network architecture facilitates the integration with any other higher-complexity networks, at a data pre- or postprocessing stage. We validate this FNN on naturally periodic smooth functions and on piecewise continuous periodic functions. We showcase the use of this FNN for modeling or solvi…
We explore in detail a method to solve ordinary differential equations using feedforward neural networks. We prove a specific loss function, which does not require knowledge of the exact solution, to be a suitable standard metric to evaluate neural networks' performance. Neural networks are shown to be proficient at approximating continuous solutions within their training domains. We illustrate neural networks' ability to outperform traditional standard numerical techniques. Training is thoroughly examined and three universal phases are found: (i) a prior tangent adjustment, (ii) a curvature f…
Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Julia Computing；Massachusetts Institute of Technology（麻省理工学院）；University of Maryland Baltimore（马里兰大学巴尔的摩分校）；Pumas AI
Democratization of machine learning requires architectures that automatically adapt to new problems. Neural Differential Equations (NDEs) have emerged as a popular modeling framework by removing the need for ML practitioners to choose the number of layers in a recurrent model. While we can control the computational cost by choosing the number of layers in standard architectures, in NDEs the number of neural network evaluations for a forward pass can depend on the number of steps of the adaptive ODE solver. But, can we force the NDE to learn the version with the least steps while not increasing…
University of Alaska Fairbanks（阿拉斯加大学费尔班克斯分校）；Oak Ridge National Laboratory（橡树岭国家实验室）
We propose a novel deep learning (DL) approach to solve one-dimensional non-linear elliptic, parabolic, and hyperbolic problems on graphs. A system of physics-informed neural network (PINN) models is used to solve the differential equations, by assigning each PINN model to a specific edge of the graph. Kirkhoff-Neumann (KN) nodal conditions are imposed in a weak form by adding a penalization term to the training loss function. Through the penalization term that imposes the KN conditions, PINN models associated with edges that share a node coordinate with each other to ensure continuity of the …
We motivate the use of neural networks for the construction of numerical solutions to differential equations. We prove that there exists a feed-forward neural network that can arbitrarily minimise an objective function that is zero at the solution of Poisson's equation, allowing us to guarantee that neural network solution estimates can get arbitrarily close to the exact solutions. We also show how these estimates can be appreciably enhanced through various strategies, in particular through the construction of error correction networks, for which we propose a general method. We conclude by pro…
John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Neural networks have been identified as powerful tools for the study of complex systems. A noteworthy example is the neural network differential equation (NN DE) solver, which can provide functional approximations to the solutions of a wide variety of differential equations. Such solvers produce robust functional expressions, are well suited for further manipulations on the quantities of interest (for example, taking derivatives), and capable of leveraging the modern advances in parallelization and computing power. However, there is a lack of work on the role precise error quantification can p…
Technical University Munich（慕尼黑工业大学）；École normale supérieure, Paris（巴黎高等师范学院）；deepc GmbH, Munich（慕尼黑deepc有限公司）；Helmholtz Zentrum München（慕尼黑亥姆霍兹研究中心）；University of Zurich（苏黎世大学）
Fast and accurate solutions of time-dependent partial differential equations (PDEs) are of pivotal interest to many research fields, including physics, engineering, and biology. Generally, implicit/semi-implicit schemes are preferred over explicit ones to improve stability and correctness. However, existing semi-implicit methods are usually iterative and employ a general-purpose solver, which may be sub-optimal for a specific class of PDEs. In this paper, we propose a neural solver to learn an optimal iterative scheme in a data-driven fashion for any class of PDEs. Specifically, we modify a si…
