The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We will also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50-100x across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs}.
Improved physics-informed neural network in mitigating gradient related failures
College of Applied Mathematics, Chengdu University of Information Technology（成都信息工程大学应用数学学院）；College of Mathematics, Sichuan University（四川大学数学学院）；School of Sciences, Southwest Petroleum University（西南石油大学理学院）
Physics-informed neural networks (PINN) have recently emerged as a promising application of deep learning in a wide range of engineering and scientific problems based on partial differential equation (PDE) models. However, evidence shows that PINN training by gradient descent displays pathologies that often prevent convergence when solving PDEs with irregular solutions. In this paper, we propose the use of a particle swarm optimization (PSO) approach to train PINNs. The resulting PSO-PINN algorithm not only mitigates the undesired behaviors of PINNs trained with standard gradient descent but a…
In this paper, we propose a refinement strategy to the well-known Physics-Informed Neural Networks (PINNs) for solving partial differential equations (PDEs) based on the concept of Optimal Transport (OT). Conventional black-box PINNs solvers have been found to suffer from a host of issues: spectral bias in fully-connected architectures, unstable gradient pathologies, as well as difficulties with convergence and accuracy. Current network training strategies are agnostic to dimension sizes and rely on the availability of powerful computing resources to optimize through a large number of collocat…
On the training of physics-informed neural operators for solving parametric partial differential equations
College of Civil Engineering, Tongji University（同济大学土木工程学院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Institute for Foundations of Data Science, Yale University（耶鲁大学数据科学基础研究所）
We propose characteristics-informed neural networks (CINN), a simple and efficient machine learning approach for solving forward and inverse problems involving hyperbolic PDEs. Like physics-informed neural networks (PINN), CINN is a meshless machine learning solver with universal approximation capabilities. Unlike PINN, which enforces a PDE softly via a multi-part loss function, CINN encodes the characteristics of the PDE in a general-purpose deep neural network by adding a characteristic layer. This neural network is trained with the usual MSE data-fitting regression loss and does not require…
WarpPINN-fibers: improved cardiac strain estimation from cine-MR with physics-informed neural networks
Millennium Institute for Intelligent Healthcare Engineering, iHEALTH；Department of Mechanical；Metallurgical Engineering, School of Engineering, Pontificia Universidad Cat\'olica de Chile, Santiago, Chile；Institute for Biological；Medical Engineering, Schools of Engineering, Medicine
Florida Atlantic University（佛罗里达大西洋大学）；Harbor Branch Oceanographic Institute, Florida Atlantic University（佛罗里达大西洋大学海港分校海洋学研究所）
According to the National Academies, a weekly forecast of velocity, vertical structure, and duration of the Loop Current (LC) and its eddies is critical for understanding the oceanography and ecosystem, and for mitigating outcomes of anthropogenic and natural disasters in the Gulf of Mexico (GoM). However, this forecast is a challenging problem since the LC behaviour is dominated by long-range spatial connections across multiple timescales. In this paper, we extend spatiotemporal predictive learning, showing its effectiveness beyond video prediction, to a 4D model, i.e., a novel Physics-inform…
SPARC-Net: A Spectral, Causality-Aware, and Hard-Constrained Physics-Informed Architecture for Stiff and Shock-Dominated Partial Differential Equations
Mitigating Gradient Pathology in PINNs through Aligned Constraint
Department of Information Science and Engineering, KTH Royal Institute of Technology, Stockholm, Sweden（信息科学与工程系，皇家理工学院，斯德哥尔摩，瑞典）；School of Advanced Manufacturing and Robotics, Peking University, Beijing, China（先进制造与机器人学院，北京大学，北京，中国）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China（先进技术学院，西安交通大学利物浦大学，苏州，中国）；Department of AI, School of Engineering, Westlake University, Hangzhou, China（人工智能系，工程学院，西湖大学，杭州，中国）
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Google Brain（谷歌大脑）；Google Inc（谷歌公司）
In this work we propose a novel data-driven, real-time power system voltage control method based on the physics-informed guided meta evolutionary strategy (ES). The main objective is to quickly provide an adaptive control strategy to mitigate the fault-induced delayed voltage recovery (FIDVR) problem. Reinforcement learning methods have been developed for the same or similar challenging control problems, but they suffer from training inefficiency and lack of robustness for "corner or unseen" scenarios. On the other hand, extensive physical knowledge has been developed in power systems but litt…
Neural networks can be used to learn the solution of partial differential equations (PDEs) on arbitrary domains without requiring a computational mesh. Common approaches integrate differential operators in training neural networks using a structured loss function. The most common training algorithm for neural networks is backpropagation which relies on the gradient of the loss function with respect to the parameters of the network. In this work, we characterize the difficulty of training neural networks on physics by investigating the impact of differential operators in corrupting the back pro…
NewPINNs: Physics-Informing Neural Networks Using Conventional Solvers for Partial Differential Equations
Department of Mathematics, University of California Riverside（加州大学河滨分校数学系）；Department of Mechanical Engineering, University of California Riverside（加州大学河滨分校机械工程系）；German Innovation Center, Procter & Gamble Service GmbH（普罗cter与 gamble 服务德国创新中心）
Enhancing Corpus Callosum Segmentation in Fetal MRI via Pathology-Informed Domain Randomization
Univeristat Politècnica de Catalunya, Barcelona, Spain（巴塞罗那理工大学）；Department of Radiology, Lausanne University Hospital and University of Lausanne (UNIL), Lausanne, Switzerland（拉索恩大学医院放射科）；University of Lausanne (UNIL), Lausanne, Switzerland（洛桑大学）；CIBM Center for Biomedical Imaging, Lausanne, Switzerland（瑞士拉索恩生物医学成像中心）；BCNatal Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de Déu), Universitat de Barcelona, Barcelona, Spain（巴塞罗那大学BCNatal胎儿医学研究中心）；Department Woman-Mother-Child, CHUV, Lausanne, Switzerland（瑞士拉索恩大学医院妇女-母亲-儿童部门）
University of Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
Climate change increases the number of extreme weather events (wind and snowstorms, heavy rains, wildfires) that compromise power system reliability and lead to multiple equipment failures. Real-time and accurate detecting of potential line failures is the first step to mitigating the extreme weather impact and activating emergency controls. Power balance equations nonlinearity, increased uncertainty in generation during extreme events, and lack of grid observability compromise the efficiency of traditional data-driven failure detection methods. At the same time, modern problem-oblivious machi…
Evolvable Psychology Informed Neural Network for Memory Behavior Modeling
National Engineering Research Center of Educational Big Data, Central China Normal University（教育大数据国家工程研究中心，中央财经大学）；Faculty of Artificial Intelligence in Education, Central China Normal University（教育人工智能学院，中央财经大学）
University of Maryland, College Park（马里兰大学帕克分校）；IBM Research（IBM研究院）
Federated learning has arisen as a mechanism to allow multiple participants to collaboratively train a model without sharing their data. In these settings, participants (workers) may not trust each other fully; for instance, a set of competitors may collaboratively train a machine learning model to detect fraud. The workers provide local gradients that a central server uses to update a global model. This global model can be corrupted when Byzantine workers send malicious gradients, which necessitates robust methods for aggregating gradients that mitigate the adverse effects of Byzantine inputs…
There has been a recent surge in adversarial attacks on deep learning based automatic speech recognition (ASR) systems. These attacks pose new challenges to deep learning security and have raised significant concerns in deploying ASR systems in safety-critical applications. In this work, we introduce WaveGuard: a framework for detecting adversarial inputs that are crafted to attack ASR systems. Our framework incorporates audio transformation functions and analyses the ASR transcriptions of the original and transformed audio to detect adversarial inputs. We demonstrate that our defense framewor…
Physics-informed neural particle flow for the Bayesian update step
Department of Control for Transportation and Vehicle Systems, Faculty of Transportation Engineering and Vehicle Engineering, Budapest University of Technology and Economics（交通运输与车辆系统控制系，交通运输工程与车辆工程学院，布达佩斯技术与经济大学）
LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization
Stanford University School of Engineering（斯坦福大学工程学院）；Stanford University School of Medicine（斯坦福大学医学院）；Stanford University Department of Statistics（斯坦福大学统计学系）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
PiDR: Physics-Informed Inertial Dead Reckoning for Autonomous Platforms
Autonomous Navigation and Sensor Fusion Lab (ANSFL)（自主导航与传感器融合实验室（ANSFL））；Hatter Department of Marine Technologies（海洋技术系）；Charney School of Marine Sciences（海洋科学学院）；University of Haifa（海法大学）
Physics-Informed Heterogeneous Graph Neural Networks for DC Blocker Placement
Argonne National Laboratory（阿贡国家实验室）；Oak Ridge National Laboratory（橡树岭国家实验室）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；University of California, Berkeley（加州大学伯克利分校）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
