Black-box optimization is a powerful approach for discovering global optima in noisy and expensive black-box functions, a problem widely encountered in real-world scenarios. Recently, there has been a growing interest in leveraging domain knowledge to enhance the efficacy of machine learning methods. Partial Differential Equations (PDEs) often provide an effective means for elucidating the fundamental principles governing the black-box functions. In this paper, we propose PINN-BO, a black-box optimization algorithm employing Physics-Informed Neural Networks that integrates the knowledge from Partial Differential Equations (PDEs) to improve the sample efficiency of the optimization. We analyze the theoretical behavior of our algorithm in terms of regret bound using advances in NTK theory and prove that the use of the PDE alongside the black-box function evaluations, PINN-BO leads to a tighter regret bound. We perform several experiments on a variety of optimization tasks and show that our algorithm is more sample-efficient compared to existing methods.
This paper presents NSGA-PINN, a multi-objective optimization framework for effective training of Physics-Informed Neural Networks (PINNs). The proposed framework uses the Non-dominated Sorting Genetic Algorithm (NSGA-II) to enable traditional stochastic gradient optimization algorithms (e.g., ADAM) to escape local minima effectively. Additionally, the NSGA-II algorithm enables satisfying the initial and boundary conditions encoded into the loss function during physics-informed training precisely. We demonstrate the effectiveness of our framework by applying NSGA-PINN to several ordinary and p…
Technical University of Munich（慕尼黑工业大学）；Pontificia Universidad Católica de Chile（智利天主教大学）；Harvard University（哈佛大学）
Physics-Informed Neural Networks (PINNs) are gaining popularity as a method for solving differential equations. While being more feasible in some contexts than the classical numerical techniques, PINNs still lack credibility. A remedy for that can be found in Uncertainty Quantification (UQ) which is just beginning to emerge in the context of PINNs. Assessing how well the trained PINN complies with imposed differential equation is the key to tackling uncertainty, yet there is lack of comprehensive methodology for this task. We propose a framework for UQ in Bayesian PINNs (B-PINNs) that incorpor…
Dalian University of Technology（大连理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；JD Explore Academy（京东探索研究院）
Physics-informed neural networks (PINNs) provide a deep learning framework for numerically solving partial differential equations (PDEs), and have been widely used in a variety of PDE problems. However, there still remain some challenges in the application of PINNs: 1) the mechanism of PINNs is unsuitable (at least cannot be directly applied) to exploiting a small size of (usually very few) extra informative samples to refine the networks; and 2) the efficiency of training PINNs often becomes low for some complicated PDEs. In this paper, we propose the generative adversarial physics-informed n…
Deep learning-based surrogate modeling is becoming a promising approach for learning and simulating dynamical systems. Deep-learning methods, however, find very challenging learning stiff dynamics. In this paper, we develop DAE-PINN, the first effective deep-learning framework for learning and simulating the solution trajectories of nonlinear differential-algebraic equations (DAE), which present a form of infinite stiffness and describe, for example, the dynamics of power networks. Our DAE-PINN bases its effectiveness on the synergy between implicit Runge-Kutta time-stepping schemes (designed …
Adaptive Interface-PINNs (AdaI-PINNs): An Efficient Physics-informed Neural Networks Framework for Interface Problems
Department of Civil Engineering, Indian Institute of Technology, Madras, Chennai 600036, Tamil Nadu, India（印度理工学院马德拉斯分校土木工程系）；Atmospheric, Earth and Energy Division, Lawrence Livermore National Laboratory, Livermore, California 94551, USA（劳伦斯利弗莫尔国家实验室大气、地球与能源部）；Institute of Civil Engineering, École Polytechnique Fédérale de Lausanne, Station 18-1015, Lausanne, Switzerland（洛桑联邦理工学院土木工程学院）
xLSTM-PINN: Memory-Gated Spectral Remodeling for Physics-Informed Learning
Nanophotonics and Biophotonics Key Laboratory of Jilin Province, School of Physics, Changchun University of Science and Technology（吉林省纳米光子与生物光子重点实验室，物理学院，长春理工大学）
SRM Institute of Science and Technology（SRM科学技术研究所）；University of Utah（犹他大学）
We present a new technique for the accelerated training of physics-informed neural networks (PINNs): discretely-trained PINNs (DT-PINNs). The repeated computation of partial derivative terms in the PINN loss functions via automatic differentiation during training is known to be computationally expensive, especially for higher-order derivatives. DT-PINNs are trained by replacing these exact spatial derivatives with high-order accurate numerical discretizations computed using meshless radial basis function-finite differences (RBF-FD) and applied via sparse-matrix vector multiplication. The use o…
Human Systems Laboratory, MIT（麻省理工学院人类系统实验室）；IBM Research（IBM研究院）；MIT Lincoln Laboratory（麻省理工学院林肯实验室）
Climate models project an uncertainty range of possible warming scenarios from 1.5 to 5 degree Celsius global temperature increase until 2100, according to the CMIP6 model ensemble. Climate risk management and infrastructure adaptation requires the accurate quantification of the uncertainties at the local level. Ensembles of high-resolution climate models could accurately quantify the uncertainties, but most physics-based climate models are computationally too expensive to run as ensemble. Recent works in physics-informed neural networks (PINNs) have combined deep learning and the physical sci…
Partition of Unity Physics-Informed Neural Networks (POU-PINNs): An Unsupervised Framework for Physics-Informed Domain Decomposition and Mixtures of Experts
The University of Texas at El Paso（德克萨斯大学埃尔帕索分校）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；Clarkson University（克拉克森大学）；Texas A&M University in Kingsville（德克萨斯农工大学金斯维尔分校）
Improved physics-informed neural network in mitigating gradient related failures
College of Applied Mathematics, Chengdu University of Information Technology（成都信息工程大学应用数学学院）；College of Mathematics, Sichuan University（四川大学数学学院）；School of Sciences, Southwest Petroleum University（西南石油大学理学院）
Minzu University of China（中央民族大学）；North China University of Technology（北方工业大学）
As a typical application of deep learning, physics-informed neural network (PINN) {has been} successfully used to find numerical solutions of partial differential equations (PDEs), but how to improve the limited accuracy is still a great challenge for PINN. In this work, we introduce a new method, symmetry-enhanced physics informed neural network (SPINN) where the invariant surface conditions induced by the Lie symmetries or non-classical symmetries of PDEs are embedded into the loss function in PINN, to improve the accuracy of PINN for solving the forward and inverse problems of PDEs. We test…
National Innovation Institute of Defense Technology, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）
Physics-informed neural networks (PINNs) have been widely used to solve various scientific computing problems. However, large training costs limit PINNs for some real-time applications. Although some works have been proposed to improve the training efficiency of PINNs, few consider the influence of initialization. To this end, we propose a New Reptile initialization based Physics-Informed Neural Network (NRPINN). The original Reptile algorithm is a meta-learning initialization method based on labeled data. PINNs can be trained with less labeled data or even without any labeled data by adding p…
PTL-PINNs: Perturbation-Guided Transfer Learning with Physics- Informed Neural Networks for Nonlinear Systems
Department of Earth Science and Engineering, Imperial College（帝国理工学院地球科学与工程系）；Harvard John A. Paulson School Of Engineering And Applied Sciences（哈佛大学约翰·A·保罗森工程与应用科学学院）
Structural failures are often caused by catastrophic events such as earthquakes and winds. As a result, it is crucial to predict dynamic stress distributions during highly disruptive events in real time. Currently available high-fidelity methods, such as Finite Element Models (FEMs), suffer from their inherent high complexity. Therefore, to reduce computational cost while maintaining accuracy, a Physics Informed Neural Network (PINN), PINN-Stress model, is proposed to predict the entire sequence of stress distribution based on Finite Element simulations using a partial differential equation (P…
Brown University（布朗大学）；Xiamen University（厦门大学）；Dalian University of Technology（大连理工大学）
Despite the significant progress over the last 50 years in simulating flow problems using numerical discretization of the Navier-Stokes equations (NSE), we still cannot incorporate seamlessly noisy data into existing algorithms, mesh-generation is complex, and we cannot tackle high-dimensional problems governed by parametrized NSE. Moreover, solving inverse flow problems is often prohibitively expensive and requires complex and expensive formulations and new computer codes. Here, we review flow physics-informed learning, integrating seamlessly data and mathematical models, and implementing the…
Physics-informed neural networks (PINNs) as a means of discretizing partial differential equations (PDEs) are garnering much attention in the Computational Science and Engineering (CS&E) world. At least two challenges exist for PINNs at present: an understanding of accuracy and convergence characteristics with respect to tunable parameters and identification of optimization strategies that make PINNs as efficient as other computational science tools. The cost of PINNs training remains a major challenge of Physics-informed Machine Learning (PiML) - and, in fact, machine learning (ML) in general…
Physics-informed neural networks (PINN) have recently emerged as a promising application of deep learning in a wide range of engineering and scientific problems based on partial differential equation (PDE) models. However, evidence shows that PINN training by gradient descent displays pathologies that often prevent convergence when solving PDEs with irregular solutions. In this paper, we propose the use of a particle swarm optimization (PSO) approach to train PINNs. The resulting PSO-PINN algorithm not only mitigates the undesired behaviors of PINNs trained with standard gradient descent but a…
National Innovation Institute of Defense Technology, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）
Physics-Informed Neural Networks (PINNs) can be regarded as general-purpose PDE solvers, but it might be slow to train PINNs on particular problems, and there is no theoretical guarantee of corresponding error bounds. In this manuscript, we propose a variant called Prior Dictionary based Physics-Informed Neural Networks (PD-PINNs). Equipped with task-dependent dictionaries, PD-PINNs enjoy enhanced representation power on the tasks, which helps to capture features provided by dictionaries so that the proposed neural networks can achieve faster convergence in the process of training. In various …
Physics-informed neural networks (PINNs) have recently received much attention due to their capabilities in solving both forward and inverse problems. For training a deep neural network associated with a PINN, one typically constructs a total loss function using a weighted sum of different loss terms and then tries to minimize that. This approach often becomes problematic for solving stiff equations since it cannot consider adaptive increments. Many studies reported the poor performance of the PINN and its challenges in simulating stiff chemical active issues with administering conditions of s…
School of Data Science（数据科学学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Department of Computer Science（计算机科学系）；Northwestern University（西北大学）；School of Science, Great Bay University（大湾大学科学学院）；Department of Mathematics（数学系）
Physics-Informed Neural Networks (PINNs) have recently been proposed to solve scientific and engineering problems, where physical laws are introduced into neural networks as prior knowledge. With the embedded physical laws, PINNs enable the estimation of critical parameters, which are unobservable via physical tools, through observable variables. For example, Power Electronic Converters (PECs) are essential building blocks for the green energy transition. PINNs have been applied to estimate the capacitance, which is unobservable during PEC operations, using current and voltage, which can be ob…
Multifidelity simulation methodologies are often used in an attempt to judiciously combine low-fidelity and high-fidelity simulation results in an accuracy-increasing, cost-saving way. Candidates for this approach are simulation methodologies for which there are fidelity differences connected with significant computational cost differences. Physics-informed Neural Networks (PINNs) are candidates for these types of approaches due to the significant difference in training times required when different fidelities (expressed in terms of architecture width and depth as well as optimization criteria…
Physics informed neural networks approximate solutions of PDEs by minimizing pointwise residuals. We derive rigorous bounds on the error, incurred by PINNs in approximating the solutions of a large class of linear parabolic PDEs, namely Kolmogorov equations that include the heat equation and Black-Scholes equation of option pricing, as examples. We construct neural networks, whose PINN residual (generalization error) can be made as small as desired. We also prove that the total $L^2$-error can be bounded by the generalization error, which in turn is bounded in terms of the training error, prov…
Scale-PINN: Learning Efficient Physics-Informed Neural Networks Through Sequential Correction
Institute of High Performance Computing（高性能计算研究所）；Centre for Frontier AI Research（前沿人工智能研究中心）；School of Mechanical Engineering（机械工程学院）；College of Computing and Data Science（计算与数据科学学院）
Brown University（布朗大学）；Pacific Northwest National Laboratory（西北太平洋国家实验室）
We propose a Bayesian physics-informed neural network (B-PINN) to solve both forward and inverse nonlinear problems described by partial differential equations (PDEs) and noisy data. In this Bayesian framework, the Bayesian neural network (BNN) combined with a PINN for PDEs serves as the prior while the Hamiltonian Monte Carlo (HMC) or the variational inference (VI) could serve as an estimator of the posterior. B-PINNs make use of both physical laws and scattered noisy measurements to provide predictions and quantify the aleatoric uncertainty arising from the noisy data in the Bayesian framewo…
AQ-PINNs: Attention-Enhanced Quantum Physics-Informed Neural Networks for Carbon-Efficient Climate Modeling
SVKM’s Dwarkadas J. Sanghvi College of Engineering（SVKM德瓦卡达斯·J·桑格维工程学院）；New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校）；NYUAD Research Institute（纽约大学阿布扎比分校研究院）；The Maersk Mc-Kinney Moller Institute（马士基麦克-金尼·莫勒研究所）；University of Southern Denmark（南丹麦大学）
FO-PINNs: A First-Order formulation for Physics Informed Neural Networks
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；NVIDIA（英伟达）；University of California Davis（加州大学戴维斯分校）；Illinois Institute of Technology（伊利诺伊理工学院）
CASUS - Center for Advanced System Understanding（先进系统理解中心）；Helmholtz-Zentrum Dresden-Rossendorf e.V. (HZDR)（亥姆霍兹德累斯顿罗森多夫研究中心）
We present a novel class of approximations for variational losses, being applicable for the training of physics-informed neural nets (PINNs). The loss formulation reflects classic Sobolev space theory for partial differential equations and their weak formulations. The loss computation rests on an extension of Gauss-Legendre cubatures, we term Sobolev cubatures, replacing automatic differentiation (A.D.). We prove the runtime complexity of training the resulting Soblev-PINNs (SC-PINNs) to be less than required by PINNs relying on A.D. On top of one-to-two order of magnitude speed-up the SC-PINN…
Solving Euler equations with Multiple Discontinuities via Separation-Transfer Physics-Informed Neural Networks
Graduate School, China Academy of Engineering Physics（中国工程物理研究院研究生院）；China Academy of Engineering Physics（中国工程物理研究院）；Zhejiang Institute of Modern Physics（浙江现代物理研究所）；School of Physics, Zhejiang University（浙江大学物理学院）；Beijing Computational Science Research Center（北京计算科学研究院）
Physics-Informed Neural Networks (PINNs) offer a promising approach to solving differential equations and, more generally, to applying deep learning to problems in the physical sciences. We adopt a recently developed transfer learning approach for PINNs and introduce a multi-head model to efficiently obtain accurate solutions to nonlinear systems of ordinary differential equations with random potentials. In particular, we apply the method to simulate stochastic branched flows, a universal phenomenon in random wave dynamics. Finally, we compare the results achieved by feed forward and GAN-based…
Conformal mapping based Physics-informed neural networks for designing neutral inclusions
Department of Mathematical Sciences, Korea Advanced Institute of Science and Technology（韩国科学技术院数学科学系）；Department of Artificial Intelligence, Chung-Ang University（Chung-Ang 大学人工智能系）
