Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of "stiff" PDEs. In this paper, we propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually, so the neural network learns autonomously which regions of the solution are difficult and is forced to focus on them. The self-adaptation weights specify a soft multiplicative soft attention mask, which is reminiscent of similar mechanisms used in computer vision. The basic idea behind these SA-PINNs is to make the weights increase as the corresponding losses increase, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights. In addition, we show how to build a continuous map of self-adaptive weights using Gaussian Process regression, which allows the use of stochastic gradient descent in problems where conventional gradient descent is not enough to produce accurate solutions. Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error, while using a smaller number of training epochs.
Brown University（布朗大学）；Massachusetts Institute of Technology（麻省理工学院）；Pacific Northwest National Laboratory（太平洋西北国家实验室）
We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each layer (layer-wise) and for every neuron (neuron-wise) separately, and then optimizing it using a variant of stochastic gradient descent algorithm. In order to further increase the training speed, an activation slope based slope recovery term is added in the loss function, which furt…
Adaptively trained Physics-informed Radial Basis Function Neural Networks for Solving Multi-asset Option Pricing Problems
College of Mathematics and Statistics, Northwest Normal University（西北师范大学数学与统计学院）；Department of Mathematics, City University of Hong Kong（香港城市大学数学系）；Department of Information Technology, Uppsala University（乌普萨拉大学信息科技系）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；NVIDIA（英伟达）
In this paper, we propose the Adaptive Physics-Informed Neural Networks (APINNs) for accurate and efficient simulation-free Bayesian parameter estimation via Markov-Chain Monte Carlo (MCMC). We specifically focus on a class of parameter estimation problems for which computing the likelihood function requires solving a PDE. The proposed method consists of: (1) constructing an offline PINN-UQ model as an approximation to the forward model; and (2) refining this approximate model on the fly using samples generated from the MCMC sampler. The proposed APINN method constantly refines this approximat…
Adaptive training methods for Physics-informed neural network (PINN) require dedicated constructions of the distribution of weights assigned at each training sample. To efficiently seek such an optimal weight distribution is not a simple task and most existing methods choose the adaptive weights based on approximating the full distribution or the maximum of residuals. In this paper, we show that the bottleneck in the adaptive choice of samples for training efficiency is the behavior of the tail distribution of the numerical residual. Thus, we propose the Residual-Quantile Adjustment (RQA) meth…
University of Southern California（南加州大学）；Collaboratory for Advanced Computing and Simulations（高级计算与模拟协作实验室）
We have developed a novel differential equation solver software called PND based on the physics-informed neural network for molecular dynamics simulators. Based on automatic differentiation technique provided by Pytorch, our software allows users to flexibly implement equation of atom motions, initial and boundary conditions, and conservation laws as loss function to train the network. PND comes with a parallel molecular dynamics (MD) engine in order for users to examine and optimize loss function design, and different conservation laws and boundary conditions, and hyperparameters, thereby acc…
An Imbalanced Learning-based Sampling Method for Physics-informed Neural Networks
The Fields Institute for Research in Mathematical Sciences（数学科学菲尔兹研究所）；Pennsylvania State University（宾夕法尼亚州立大学）；Data Science Research Center, Duke Kunshan University（昆山杜克大学数据科学研究中心）；Zu Chongzhi Center for Mathematics and Computational Sciences, Duke Kunshan University（昆山杜克大学祖冲之数学与计算科学中心）
Brown University（布朗大学）；John Hopkins University（约翰斯·霍普金斯大学）；Air Force Research Laboratory（美国空军研究实验室）
We introduce an optimized physics-informed neural network (PINN) trained to solve the problem of identifying and characterizing a surface breaking crack in a metal plate. PINNs are neural networks that can combine data and physics in the learning process by adding the residuals of a system of Partial Differential Equations to the loss function. Our PINN is supervised with realistic ultrasonic surface acoustic wave data acquired at a frequency of 5 MHz. The ultrasonic surface wave data is represented as a surface deformation on the top surface of a metal plate, measured by using the method of l…
Defense Innovation Institute, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）；Human Resource Center（人力资源中心）
Learning solutions of partial differential equations (PDEs) with Physics-Informed Neural Networks (PINNs) is an attractive alternative approach to traditional solvers due to its flexibility and ease of incorporating observed data. Despite the success of PINNs in accurately solving a wide variety of PDEs, the method still requires improvements in terms of computational efficiency. One possible improvement idea is to optimize the generation of training point sets. Residual-based adaptive sampling and quasi-uniform sampling approaches have been each applied to improve the training effects of PINN…
Adaptive Physics-Informed Neural Networks with Multi-Category Feature Engineering for Hydrogen Sorption Prediction in Clays, Shales, and Coals
Faculty of Mathematics & Natural Sciences, University of Oslo（数学与自然科学学院，奥斯陆大学）；SINTEF Industry, Applied Geoscience Department（SINTEF工业公司，应用地球科学部门）；College of Geosciences, China University of Petroleum（地质科学学院，中国石油大学）
Multi-Fidelity Physics-Informed Neural Networks with Bayesian Uncertainty Quantification and Adaptive Residual Learning for Efficient Solution of Parametric Partial Differential Equations
Department of Applied Mathematics and Computer Science (DTU Compute)（应用数学与计算机科学系（DTU Compute））；Technical University of Denmark（丹麦技术大学）
Physics-Informed Neural Networks (PINNs) have become a kind of attractive machine learning method for obtaining solutions of partial differential equations (PDEs). Training PINNs can be seen as a semi-supervised learning task, in which only exact values of initial and boundary points can be obtained in solving forward problems, and in the whole spatio-temporal domain collocation points are sampled without exact labels, which brings training difficulties. Thus the selection of collocation points and sampling methods are quite crucial in training PINNs. Existing sampling methods include fixed an…
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；University of Utah（犹他大学）；International Computer Science Institute（国际计算机科学研究所）；University of California Berkeley（加州大学伯克利分校）
Physics-informed neural networks (PINNs) incorporate physical knowledge from the problem domain as a soft constraint on the loss function, but recent work has shown that this can lead to optimization difficulties. Here, we study the impact of the location of the collocation points on the trainability of these models. We find that the vanilla PINN performance can be significantly boosted by adapting the location of the collocation points as training proceeds. Specifically, we propose a novel adaptive collocation scheme which progressively allocates more collocation points (without increasing th…
Adaptive Interface-PINNs (AdaI-PINNs): An Efficient Physics-informed Neural Networks Framework for Interface Problems
Department of Civil Engineering, Indian Institute of Technology, Madras, Chennai 600036, Tamil Nadu, India（印度理工学院马德拉斯分校土木工程系）；Atmospheric, Earth and Energy Division, Lawrence Livermore National Laboratory, Livermore, California 94551, USA（劳伦斯利弗莫尔国家实验室大气、地球与能源部）；Institute of Civil Engineering, École Polytechnique Fédérale de Lausanne, Station 18-1015, Lausanne, Switzerland（洛桑联邦理工学院土木工程学院）
Reliable and efficient inverse analysis using physics-informed neural networks with normalized distance functions and adaptive weight tuning
Institute of Systems and Information Engineering University of Tsukuba（系统与信息工程研究所，茨ukuba大学）；Department of Civil Engineering Kyushu University（土木工程系，九州大学）
Adaptive Hard-Soft Physics-Informed Neural Networks for Robust Boundary-Constrained PDE Solving
College of Engineering and Computer Science, VinUniversity（VinUniversity工程与计算机科学学院）；Center for AI Research, VinUniversity（VinUniversity人工智能研究中心）；Group of Materials and Structures, School of Mechanical Engineering, Hanoi University of Science and Technology（河内科技大学机械工程学院材料与结构组）；Department of Mathematics and Informatics, Hanoi University of Science and Technology（河内科技大学数学与信息学系）
Energy Dissipation Rate Guided Adaptive Sampling for Physics-Informed Neural Networks: Resolving Surface-Bulk Dynamics in Allen-Cahn Systems
Department of Mathematics, The Pennsylvania State University（数学系，宾夕法尼亚州立大学）；Department of Mathematics, The Hong Kong University of Science and Technology（数学系，香港科学与技术大学）；Department of Mathematics, University of South Carolina（数学系，南卡罗来纳大学）
Adaptive Physics-informed Neural Networks: A Survey
Institute for Artificial Intelligence, University of Stuttgart（斯图加特大学人工智能研究所）；Robert Bosch GmbH（罗伯特·博世有限公司）；IMPRS-IS（国际研究生学院智能系统（IMPRS-IS））；NEC Labs Europe（NEC欧洲实验室）；SimTech（仿真技术研究所（SimTech））
An adaptive framework for the axisymmetric pulsar magnetosphere using physics-informed Kolmogorov-Arnold networks
Department of Digital Industry Technologies, School of Science, National and Kapodistrian University of Athens（数字产业技术系，科学学院，国家与卡布利安大学）；Research Center for Astronomy and Applied Mathematics, Academy of Athens（天文与应用数学研究所，雅典学院）
PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Short-term Weather Forecasting
Diversity-Aware Adaptive Collocation for Physics-Informed Neural Networks via Sparse QUBO Optimization and Hybrid Coresets
Research Center for Artificial Intelligence（人工智能研究中心）；Innopolis University（因诺波利斯大学）；Moscow Center for Advanced Studies（莫斯科先进研究中心）；Laboratory of Quantum Computing（量子计算实验室）；Center for Engineering Systems and Sciences（工程系统与科学中心）；Central University（中央大学）；The Center for Top-Level Educational Programs in AI（人工智能顶级教育项目中心）；National Medical Research Center of Oncology（肿瘤医学研究中心）
St. Mark’s School of Texas（圣马克德克萨斯学校）；University of Pennsylvania（宾夕法尼亚大学）；Brown University（布朗大学）
Deep learning has been shown to be an effective tool in solving partial differential equations (PDEs) through physics-informed neural networks (PINNs). PINNs embed the PDE residual into the loss function of the neural network, and have been successfully employed to solve diverse forward and inverse PDE problems. However, one disadvantage of the first generation of PINNs is that they usually have limited accuracy even with many training points. Here, we propose a new method, gradient-enhanced physics-informed neural networks (gPINNs), for improving the accuracy and training efficiency of PINNs.…
