Predicting outcomes and planning interactions with the physical world are long-standing goals for machine learning. A variety of such tasks involves continuous physical systems, which can be described by partial differential equations (PDEs) with many degrees of freedom. Existing methods that aim to control the dynamics of such systems are typically limited to relatively short time frames or a small number of interaction parameters. We present a novel hierarchical predictor-corrector scheme which enables neural networks to learn to understand and control complex nonlinear physical systems over long time frames. We propose to split the problem into two distinct tasks: planning and control. To this end, we introduce a predictor network that plans optimal trajectories and a control network that infers the corresponding control parameters. Both stages are trained end-to-end using a differentiable PDE solver. We demonstrate that our method successfully develops an understanding of complex physical systems and learns to control them for tasks involving PDEs such as the incompressible Navier-Stokes equations.
Weak-PDE-Net: Discovering Open-Form PDEs via Differentiable Symbolic Networks and Weak Formulation
School of Mathematical Sciences, East China Normal University（东华大学数学科学学院）；Institute of Applied Physics and Computational Mathematics（应用物理与计算数学研究所）；Institute of Artificial Intelligence, Beihang University（北航人工智能研究所）；Shanghai Zhangjiang Institute of Mathematics（上海张江数学研究所）；Academy for Advanced Interdisciplinary Studies, Northeast Normal University（东北师范大学先进交叉学科研究院）
Physics-informed Neural Networks (PINNs) have been widely used to obtain accurate neural surrogates for a system of Partial Differential Equations (PDE). One of the major limitations of PINNs is that the neural solutions are challenging to interpret, and are often treated as black-box solvers. While Symbolic Regression (SR) has been studied extensively, very few works exist which generate analytical expressions to directly perform SR for a system of PDEs. In this work, we introduce an end-to-end framework for obtaining mathematical expressions for solutions of PDEs. We use a trained PINN to ge…
Physical systems are governed by partial differential equations (PDEs). The Navier-Stokes equations describe fluid flows and are representative of nonlinear physical systems with complex spatio-temporal interactions. Fluid flows are omnipresent in nature and engineering applications, and their accurate simulation is essential for providing insights into these processes. While PDEs are typically solved with numerical methods, the recent success of machine learning (ML) has shown that ML methods can provide novel avenues of finding solutions to PDEs. ML is becoming more and more present in compu…
Technical University of Munich（慕尼黑工业大学）；LTCI, Telecom Paris, IP Paris（巴黎理工学院巴黎电信学院LTCI）；Columbia University（哥伦比亚大学）
Finding accurate solutions to partial differential equations (PDEs) is a crucial task in all scientific and engineering disciplines. It has recently been shown that machine learning methods can improve the solution accuracy by correcting for effects not captured by the discretized PDE. We target the problem of reducing numerical errors of iterative PDE solvers and compare different learning approaches for finding complex correction functions. We find that previously used learning approaches are significantly outperformed by methods that integrate the solver into the training loop and thereby a…
In this paper we propose a new model-based unsupervised learning method, called VarNet, for the solution of partial differential equations (PDEs) using deep neural networks (NNs). Particularly, we propose a novel loss function that relies on the variational (integral) form of PDEs as apposed to their differential form which is commonly used in the literature. Our loss function is discretization-free, highly parallelizable, and more effective in capturing the solution of PDEs since it employs lower-order derivatives and trains over measure non-zero regions of space-time. Given this loss functio…
Carnegie Mellon University（卡内基梅隆大学）；SU2 Foundation（SU2基金会）；Bosch Center for AI（博世人工智能中心）
Solving large complex partial differential equations (PDEs), such as those that arise in computational fluid dynamics (CFD), is a computationally expensive process. This has motivated the use of deep learning approaches to approximate the PDE solutions, yet the simulation results predicted from these approaches typically do not generalize well to truly novel scenarios. In this work, we develop a hybrid (graph) neural network that combines a traditional graph convolutional network with an embedded differentiable fluid dynamics simulator inside the network itself. By combining an actual CFD simu…
The University of Tokyo（东京大学）；KAIST（韩国科学技术院）；RIKEN（理化学研究所）
We detail a novel class of implicit neural models. Leveraging time-parallel methods for differential equations, Multiple Shooting Layers (MSLs) seek solutions of initial value problems via parallelizable root-finding algorithms. MSLs broadly serve as drop-in replacements for neural ordinary differential equations (Neural ODEs) with improved efficiency in number of function evaluations (NFEs) and wall-clock inference time. We develop the algorithmic framework of MSLs, analyzing the different choices of solution methods from a theoretical and computational perspective. MSLs are showcased in long…
Solved in Unit Domain: JacobiNet for Differentiable Coordinate-Transformed PINNs
Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology（香港科技大学机械与航空航天工程系）；Department of Mechanics & Engineering, College Architecture & Environment, Sichuan University（四川大学力学与工程学院）；School of Mechanical Engineering and Automation, Beihang University（北京航空航天大学机械工程与自动化学院）；Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology（香港科技大学土木与环境工程系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；West China Biomedical Big Data Center, West China Hospital, Sichuan University（四川大学西昌生物医学大数据中心，西昌医院）
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Providing fast and accurate solutions to partial differential equations is a problem of continuous interest to the fields of applied mathematics and physics. With the recent advances in machine learning, the adoption learning techniques in this domain is being eagerly pursued. We build upon earlier works on linear and homogeneous PDEs, and develop convolutional deep neural networks that can accurately solve nonlinear and non-homogeneous equations without the need for labeled data. The architecture of these networks is readily accessible for scientific disciplines who deal with PDEs and know th…
University of Southern California（南加州大学）；Department of Computer Science, University of Southern California（南加州大学计算机科学系）
While physics conveys knowledge of nature built from an interplay between observations and theory, it has been considered less importantly in deep neural networks. Especially, there are few works leveraging physics behaviors when the knowledge is given less explicitly. In this work, we propose a novel architecture called Differentiable Physics-informed Graph Networks (DPGN) to incorporate implicit physics knowledge which is given from domain experts by informing it in latent space. Using the concept of DPGN, we demonstrate that climate prediction tasks are significantly improved. Besides the e…
Learning, Solving and Optimizing PDEs with TensorGalerkin: an efficient high-performance Galerkin assembly algorithm
ETH Zurich, Switzerland（苏黎世联邦理工学院，瑞士）；Imperial College London, UK（伦敦帝国学院，英国）；Northeastern University, USA（东北大学，美国）；Renmin University of China, China（中国人民大学，中国）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We design a physics-aware auto-encoder to specifically reduce the dimensionality of solutions arising from convection-dominated nonlinear physical systems. Although existing nonlinear manifold learning methods seem to be compelling tools to reduce the dimensionality of data characterized by a large Kolmogorov n-width, they typically lack a straightforward mapping from the latent space to the high-dimensional physical space. Moreover, the realized latent variables are often hard to interpret. Therefore, many of these methods are often dismissed in the reduced order modeling of dynamical systems…
Numerous physics theories are rooted in partial differential equations (PDEs). However, the increasingly intricate physics equations, especially those that lack analytic solutions or closed forms, have impeded the further development of physics. Computationally solving PDEs by classic numerical approaches suffers from the trade-off between accuracy and efficiency and is not applicable to the empirical data generated by unknown latent PDEs. To overcome this challenge, we present KoopmanLab, an efficient module of the Koopman neural operator family, for learning PDEs without analytic solutions o…
University of Bonn（波恩大学）；Delft University of Technology（代尔夫特理工大学）；RWTH Aachen University（亚琛工业大学）
Partial Differential Equations (PDEs) are notoriously difficult to solve. In general, closed-form solutions are not available and numerical approximation schemes are computationally expensive. In this paper, we propose to approach the solution of PDEs based on a novel technique that combines the advantages of two recently emerging machine learning based approaches. First, physics-informed neural networks (PINNs) learn continuous solutions of PDEs and can be trained with little to no ground truth data. However, PINNs do not generalize well to unseen domains. Second, convolutional neural network…
Deep Forest Sciences Inc.（深林科学公司）；Carnegie Mellon University（卡内基梅隆大学）
Differentiable physics provides a new approach for modeling and understanding the physical systems by pairing the new technology of differentiable programming with classical numerical methods for physical simulation. We survey the rapidly growing literature of differentiable physics techniques and highlight methods for parameter estimation, learning representations, solving differential equations, and developing what we call scientific foundation models using data and inductive priors. We argue that differentiable physics offers a new paradigm for modeling physical phenomena by combining class…
Physics-informed neural networks (PINNs) encode physical conservation laws and prior physical knowledge into the neural networks, ensuring the correct physics is represented accurately while alleviating the need for supervised learning to a great degree. While effective for relatively short-term time integration, when long time integration of the time-dependent PDEs is sought, the time-space domain may become arbitrarily large and hence training of the neural network may become prohibitively expensive. To this end, we develop a parareal physics-informed neural network (PPINN), hence decomposin…
University of Maryland, College Park（马里兰大学帕克分校）；Intel Labs（英特尔实验室）
Differentiable physics is a powerful approach to learning and control problems that involve physical objects and environments. While notable progress has been made, the capabilities of differentiable physics solvers remain limited. We develop a scalable framework for differentiable physics that can support a large number of objects and their interactions. To accommodate objects with arbitrary geometry and topology, we adopt meshes as our representation and leverage the sparsity of contacts for scalable differentiable collision handling. Collisions are resolved in localized regions to minimize …
Physics-informed neural networks (PINNs) as a means of discretizing partial differential equations (PDEs) are garnering much attention in the Computational Science and Engineering (CS&E) world. At least two challenges exist for PINNs at present: an understanding of accuracy and convergence characteristics with respect to tunable parameters and identification of optimization strategies that make PINNs as efficient as other computational science tools. The cost of PINNs training remains a major challenge of Physics-informed Machine Learning (PiML) - and, in fact, machine learning (ML) in general…
Physics informed neural networks approximate solutions of PDEs by minimizing pointwise residuals. We derive rigorous bounds on the error, incurred by PINNs in approximating the solutions of a large class of linear parabolic PDEs, namely Kolmogorov equations that include the heat equation and Black-Scholes equation of option pricing, as examples. We construct neural networks, whose PINN residual (generalization error) can be made as small as desired. We also prove that the total $L^2$-error can be bounded by the generalization error, which in turn is bounded in terms of the training error, prov…
Time-dependent Partial Differential Equations with given initial conditions are considered in this paper. New differentiation techniques of the unknown solution with respect to time variable are proposed. It is shown that the proposed techniques allow to generate accurate higher order derivatives simultaneously for a set of spatial points. The calculated derivatives can then be used for data-driven solution in different ways. An application for Physics Informed Neural Networks by the well-known DeepXDE software solution in Python under Tensorflow background framework has been presented for thr…
Gaussian process surrogate with physical law-corrected prior for multi-coupled PDEs defined on irregular geometry
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Mathematics and Statistics, Central South University（中南大学数学与统计学院）；Institute of Applied Physics and Computational Mathematics（北京应用物理与计算数学研究所）；Institute of Mathematics, Henan Academy of Sciences（河南省科学院数学研究所）
Quantum-classical physics-informed Kolmogorov-Arnold networks for PDEs
School of Petroleum Engineering, Yangtze University（扬州大学石油工程学院）；School of Computer Science, Yangtze University（扬州大学计算机科学学院）；State Key Laboratory of Low Carbon Catalysis and Carbon Dioxide Utilization (Yangtze University)（扬州大学低碳催化与二氧化碳利用国家重点实验室）；Western Research Institute, Yangtze University（扬州大学西部研究院）
Linear Partial Differential Equations (PDEs) govern the spatial-temporal dynamics of physical systems that are essential to building modern technology. When working with linear PDEs, designing a physical system for a specific outcome is difficult and costly due to slow and expensive explicit simulation of PDEs and the highly nonlinear relationship between a system's configuration and its behavior. In this work, we prove a parametric form that certain physical quantities in the Fourier domain must obey in linear PDEs, named the CZP (Constant-Zeros-Poles) framework. Applying CZP to antenna desig…
Physics informed neural networks (PINNs) are deep learning based techniques for solving partial differential equations (PDEs) encounted in computational science and engineering. Guided by data and physical laws, PINNs find a neural network that approximates the solution to a system of PDEs. Such a neural network is obtained by minimizing a loss function in which any prior knowledge of PDEs and data are encoded. Despite its remarkable empirical success in one, two or three dimensional problems, there is little theoretical justification for PINNs. As the number of data grows, PINNs generate a se…
We propose quadratic residual networks (QRes) as a new type of parameter-efficient neural network architecture, by adding a quadratic residual term to the weighted sum of inputs before applying activation functions. With sufficiently high functional capacity (or expressive power), we show that it is especially powerful for solving forward and inverse physics problems involving partial differential equations (PDEs). Using tools from algebraic geometry, we theoretically demonstrate that, in contrast to plain neural networks, QRes shows better parameter efficiency in terms of network width and de…
QCPINN: Quantum-Classical Physics-Informed Neural Networks for Solving PDEs
Computational Science and Engineering Department, Informatics Institute Istanbul Technical University（信息学院理工学院计算科学与工程系）；Scientific Computing, Rutherford Appleton Laboratory Science and Technology Facilities Council (STFC)（科学与技术设施委员会（STFC）科学计算部）
Parameterized Physics-informed Neural Networks for Parameterized PDEs
Yonsei University（延世大学）；Arizona State University（亚利桑那州立大学）；LG CNS；Texas A&M University-Commerce（德州农工大学Commerce分校）；Oregon State University（俄勒冈州立大学）；KAIST（韩国科学技术院）
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Adobe Research（奥多比研究院）；UC Berkeley（加州大学伯克利分校）
We present DiffTaichi, a new differentiable programming language tailored for building high-performance differentiable physical simulators. Based on an imperative programming language, DiffTaichi generates gradients of simulation steps using source code transformations that preserve arithmetic intensity and parallelism. A light-weight tape is used to record the whole simulation program structure and replay the gradient kernels in a reversed order, for end-to-end backpropagation. We demonstrate the performance and productivity of our language in gradient-based learning and optimization tasks on…
There has been a growing interest in the use of Deep Neural Networks (DNNs) to solve Partial Differential Equations (PDEs). Despite the promise that such approaches hold, there are various aspects where they could be improved. Two such shortcomings are (i) their computational inefficiency relative to classical numerical methods, and (ii) the non-interpretability of a trained DNN model. In this work we present ASPINN, an anisotropic extension of our earlier work called SPINN--Sparse, Physics-informed, and Interpretable Neural Networks--to solve PDEs that addresses both these issues. ASPINNs gen…
Compared with conventional numerical approaches to solving partial differential equations (PDEs), physics-informed neural networks (PINN) have manifested the capability to save development effort and computational cost, especially in scenarios of reconstructing the physics field and solving the inverse problem. Considering the advantages of parameter sharing, spatial feature extraction and low inference cost, convolutional neural networks (CNN) are increasingly used in PINN. However, some challenges still remain as follows. To adapt convolutional PINN to solve different PDEs, considerable effo…
Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale ra…
Physics-informed learning has shown to have a better generalization than learning without physical priors. However, training physics-informed deep neural networks requires some aspect of physical simulations to be written in a differentiable manner. Unfortunately, some operations and functionals commonly used in physical simulations are scattered, hard to integrate, and lack higher order derivatives which are needed in physical simulations. In this work, we present $ξ$-torch, a library of differentiable functionals for scientific simulations. Example functionals are a root finder and an initia…
TINNs: Time-Induced Neural Networks for Solving Time-Dependent PDEs
Department of Applied Mathematics, National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan（应用数学系，国立阳明交通大学，新竹30010，台湾）；Institute of Artificial Intelligence Innovation, National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan（人工智能创新研究所，国立阳明交通大学，新竹30010，台湾）；National Center for Theoretical Sciences, National Taiwan University, Taipei 10617, Taiwan（理论科学研究中心，国立台湾大学，台北10617，台湾）
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Center for Machine Intelligence and Data Science (CMINDS), Indian Institute of Technology Bombay（印度理工学院孟买分校机器智能与数据科学中心（CMINDS））
We introduce a class of Sparse, Physics-based, and partially Interpretable Neural Networks (SPINN) for solving ordinary and partial differential equations (PDEs). By reinterpreting a traditional meshless representation of solutions of PDEs we develop a class of sparse neural network architectures that are partially interpretable. The SPINN model we propose here serves as a seamless bridge between two extreme modeling tools for PDEs, namely dense neural network based methods like Physics Informed Neural Networks (PINNs) and traditional mesh-free numerical methods, thereby providing a novel mean…
Brown University（布朗大学）；NVIDIA（英伟达）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Julia Computing（朱莉娅计算公司）；Pacific Northwest National Lab（太平洋西北国家实验室）；Massachusetts Institute of Technology（麻省理工学院）
Uncertainty quantification for forward and inverse problems is a central challenge across physical and biomedical disciplines. We address this challenge for the problem of modeling subsurface flow at the Hanford Site by combining stochastic computational models with observational data using physics-informed GAN models. The geographic extent, spatial heterogeneity, and multiple correlation length scales of the Hanford Site require training a computationally intensive GAN model to thousands of dimensions. We develop a hierarchical scheme for exploiting domain parallelism, map discriminators and …
