This paper addresses the training of Neural Ordinary Differential Equations (neural ODEs), and in particular explores the interplay between numerical integration techniques, stability regions, step size, and initialization techniques. It is shown how the choice of integration technique implicitly regularizes the learned model, and how the solver's corresponding stability region affects training and prediction performance. From this analysis, a stability-informed parameter initialization technique is introduced. The effectiveness of the initialization method is displayed across several learning benchmarks and industrial applications.
Operator-informed initialization for Fourier features physics-informed neural networks
National Center for Artificial Intelligence（国家人工智能中心）；University of Pennsylvania（宾夕法尼亚大学）；Pontificia Universidad Católica de Chile（智利天主教大学）；Millennium Institute for Intelligent Healthcare Engineering, iHEALTH（千禧智能医疗工程研究所）
National Innovation Institute of Defense Technology, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）
Physics-informed neural networks (PINNs) have been widely used to solve various scientific computing problems. However, large training costs limit PINNs for some real-time applications. Although some works have been proposed to improve the training efficiency of PINNs, few consider the influence of initialization. To this end, we propose a New Reptile initialization based Physics-Informed Neural Network (NRPINN). The original Reptile algorithm is a meta-learning initialization method based on labeled data. PINNs can be trained with less labeled data or even without any labeled data by adding p…
Reducing Oversmoothing through Informed Weight Initialization in Graph Neural Networks
School of Electrical Engineering and Computer Science, National Technical University of Athens（雅典国立技术大学电气与计算机工程学院）；National Center for Scientific Research “Demokritos”（德谟克利特国家科学研究中心）
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY)（南洋理工大学-不列颠哥伦比亚大学老年活跃生活联合卓越研究中心）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）；Alibaba Group（阿里巴巴集团）
Proper initialization is crucial to the optimization and the generalization of neural networks. However, most existing neural recommendation systems initialize the user and item embeddings randomly. In this work, we propose a new initialization scheme for user and item embeddings called Laplacian Eigenmaps with Popularity-based Regularization for Isolated Data (LEPORID). LEPORID endows the embeddings with information regarding multi-scale neighborhood structures on the data manifold and performs adaptive regularization to compensate for high embedding variance on the tail of the data distribut…
Joint Initialization of Flux Networks and Effective Multiplication Factor for Physics-Informed Neural Networks Solving Neutron Diffusion Problems
School of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院）；Center for Scientific Intelligence Innovation, University of Science and Technology of China（中国科学技术大学科学智能创新中心）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；Texas A & M University（得克萨斯农工大学）；University of Notre Dame（圣母大学）
In this work a novel, automated process for constructing and initializing deep feed-forward neural networks based on decision trees is presented. The proposed algorithm maps a collection of decision trees trained on the data into a collection of initialized neural networks, with the structures of the networks determined by the structures of the trees. The tree-informed initialization acts as a warm-start to the neural network training process, resulting in efficiently trained, accurate networks. These models, referred to as "deep jointly-informed neural networks" (DJINN), demonstrate high pred…
Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）
In this work, we assess the ability of physics-informed neural networks (PINNs) to solve increasingly-complex coupled ordinary differential equations (ODEs). We focus on a pair of benchmarks: discretized partial differential equations and harmonic oscillators, each of which has a tunable parameter that controls its complexity. Even by varying network architecture and applying a state-of-the-art training method that accounts for "difficult" training regions, we show that PINNs eventually fail to produce correct solutions to these benchmarks as their complexity -- the number of equations and the…
University of California, Berkeley（加州大学伯克利分校）；The Ohio State University（俄亥俄州立大学）
Stochastic differential equations (SDEs) are used to describe a wide variety of complex stochastic dynamical systems. Learning the hidden physics within SDEs is crucial for unraveling fundamental understanding of these systems' stochastic and nonlinear behavior. We propose a flexible and scalable framework for training artificial neural networks to learn constitutive equations that represent hidden physics within SDEs. The proposed stochastic physics-informed neural ordinary differential equation framework (SPINODE) propagates stochasticity through the known structure of the SDE (i.e., the kno…
Informed by the basic geometry underlying feed forward neural networks, we initialize the weights of the first layer of a neural network using the linear discriminants which best distinguish individual classes. Networks initialized in this way take fewer training steps to reach the same level of training, and asymptotically have higher accuracy on training data.
A new initialization method for hidden parameters in a neural network is proposed. Derived from the integral representation of the neural network, a nonparametric probability distribution of hidden parameters is introduced. In this proposal, hidden parameters are initialized by samples drawn from this distribution, and output parameters are fitted by ordinary linear regression. Numerical experiments show that backpropagation with proposed initialization converges faster than uniformly random initialization. Also it is shown that the proposed method achieves enough accuracy by itself without ba…
Training Deep Physics-Informed Kolmogorov-Arnold Networks
organization= Department of Digital Industry Technologies, School of Science, National；Kapodistrian University of Athens , postcode= 34400 , country= Greece；organization= Department of Informatics \& Telecommunications, University of the Peloponnese , postcode= 22131 , country= Greece；organization= Department of Physics, School of Science, National；Kapodistrian University of Athens , postcode= 15784 , country= Greece
In this work, we propose a data-driven scheme to initialize the parameters of a deep neural network. This is in contrast to traditional approaches which randomly initialize parameters by sampling from transformed standard distributions. Such methods do not use the training data to produce a more informed initialization. Our method uses a sequential layer-wise approach where each layer is initialized using its input activations. The initialization is cast as an optimization problem where we minimize a combination of encoding and decoding losses of the input activations, which is further constra…
Learning dynamical systems with biochemically informed neural ordinary differential equations
Laboratory for Systems Medicine, Department of Medicine, University of Florida（系统医学实验室，医学系，佛罗里达大学）；Department of Computational Science and Philosophy, Frankfurt School of Finance and Management（计算科学与哲学系，法兰克福金融与管理学院）
Solving Oscillator Ordinary Differential Equations in the Time Domain with High Performance via Soft-constrained Physics-informed Neural Network with Small Data
College of Information Technology, Shanghai Jian Qiao University（上海建桥学院信息技术学院）；School of Urban Railway Transportation, Shanghai University of Engineering Science（上海工程技术大学城市轨道交通学院）
Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operators
Tsinghua University（清华大学）；Yau Mathematical Sciences Center, Tsinghua University（清华大学丘成桐数学科学中心）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（雁栖湖北京数学科学与应用研究院）
Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We provide analysis of these techniques and illustrate via numerical examples dramatic increases in accuracy and convergence rate for benchmarks characterizing scientific applications where DNNs are currently used, including regression problems and physics-informed neural networks …
Physics-Informed Neural Network Frameworks for the Analysis of Engineering and Biological Dynamical Systems Governed by Ordinary Differential Equations
Siena Heights University（锡耶纳高地大学）；Wayne State University（韦恩州立大学）
University of Oxford（牛津大学）；Harvard University（哈佛大学）
Solving differential equations efficiently and accurately sits at the heart of progress in many areas of scientific research, from classical dynamical systems to quantum mechanics. There is a surge of interest in using Physics-Informed Neural Networks (PINNs) to tackle such problems as they provide numerous benefits over traditional numerical approaches. Despite their potential benefits for solving differential equations, transfer learning has been under explored. In this study, we present a general framework for transfer learning PINNs that results in one-shot inference for linear systems of …
This paper presents NSGA-PINN, a multi-objective optimization framework for effective training of Physics-Informed Neural Networks (PINNs). The proposed framework uses the Non-dominated Sorting Genetic Algorithm (NSGA-II) to enable traditional stochastic gradient optimization algorithms (e.g., ADAM) to escape local minima effectively. Additionally, the NSGA-II algorithm enables satisfying the initial and boundary conditions encoded into the loss function during physics-informed training precisely. We demonstrate the effectiveness of our framework by applying NSGA-PINN to several ordinary and p…
HKUST(GZ)（香港科技大学（广州））；HKUST（香港科技大学）；Future Resilient Systems, Singapore-ETH Centre（新加坡-苏黎世联邦理工学院中心未来弹性系统）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）；Shanghai Qizhi Institute（上海期智研究院）
The order/dimension of models derived on the basis of data is commonly restricted by the number of observations, or in the context of monitored systems, sensing nodes. This is particularly true for structural systems (e.g., civil or mechanical structures), which are typically high-dimensional in nature. In the scope of physics-informed machine learning, this paper proposes a framework -- termed Neural Modal ODEs -- to integrate physics-based modeling with deep learning for modeling the dynamics of monitored and high-dimensional engineered systems. Neural Ordinary Differential Equations -- Neur…
Sandia National Laboratories（桑迪亚国家实验室）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of New Mexico（新墨西哥大学）
This paper investigates multilevel initialization strategies for training very deep neural networks with a layer-parallel multigrid solver. The scheme is based on the continuous interpretation of the training problem as a problem of optimal control, in which neural networks are represented as discretizations of time-dependent ordinary differential equations. A key goal is to develop a method able to intelligently initialize the network parameters for the very deep networks enabled by scalable layer-parallel training. To do this, we apply a refinement strategy across the time domain, that is eq…
University of Pennsylvania（宾夕法尼亚大学）；Brown University（布朗大学）
The dynamics of systems biological processes are usually modeled by a system of ordinary differential equations (ODEs) with many unknown parameters that need to be inferred from noisy and sparse measurements. Here, we introduce systems-biology informed neural networks for parameter estimation by incorporating the system of ODEs into the neural networks. To complete the workflow of system identification, we also describe structural and practical identifiability analysis to analyze the identifiability of parameters. We use the ultridian endocrine model for glucose-insulin interaction as the exam…
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Cornell University（康奈尔大学）
Neural networks (NN)-based learning algorithms are strongly affected by the choices of initialization and data distribution. Different optimization strategies have been proposed for improving the learning trajectory and finding a better optima. However, designing improved optimization strategies is a difficult task under the conventional landscape view. Here, we propose persistent neurons, a trajectory-based strategy that optimizes the learning task using information from previous converged solutions. More precisely, we utilize the end of trajectories and let the parameters explore new landsca…
Advanced Research Center, National Fusion Research Institute (NFRI)（国家聚变研究所高级研究中心）
Inspired by recent theoretical arguments, physics-informed echo state network (ESN) is discussed on the attempt to train a reservoir model absolutely in physics-informed manner. As the plainest work on such a purpose, an ODE (ordinary differential equation) approximator is designed to replicate the solution in sequence with respect to the recurrent evaluations. On the principal invariance of differential equations, the constraint in recurrence just takes shape to secure a proper regression method for the ESN-based ODE approximator. After then, the actual training process is established on the …
This paper presents a PINN training framework that employs (1) pre-training steps that accelerates and improve the robustness of the training of physics-informed neural network with auxiliary data stored in point clouds, (2) a net-to-net knowledge transfer algorithm that improves the weight initialization of the neural network and (3) a multi-objective optimization algorithm that may improve the performance of a physical-informed neural network with competing constraints. We consider the training and transfer and multi-task learning of physics-informed neural network (PINN) as multi-objective …
Physics-Informed Neural Networks (PINNs) offer a promising approach to solving differential equations and, more generally, to applying deep learning to problems in the physical sciences. We adopt a recently developed transfer learning approach for PINNs and introduce a multi-head model to efficiently obtain accurate solutions to nonlinear systems of ordinary differential equations with random potentials. In particular, we apply the method to simulate stochastic branched flows, a universal phenomenon in random wave dynamics. Finally, we compare the results achieved by feed forward and GAN-based…
Physics-Informed Neural Networks for Methane Sorption: Cross-Gas Transfer Learning, Ensemble Collapse Under Physics Constraints, and Monte Carlo Dropout Uncertainty Quantification
Environmental Geosciences, Department of Geosciences, University of Oslo（奥斯陆大学环境地质学系）；National Key Laboratory of Petroleum Resources and Engineering, China University of Petroleum (Beijing)（中国石油大学（北京）石油资源与工程国家级重点实验室）；College of Geosciences, China University of Petroleum (Beijing)（中国石油大学（北京）地质学院）；State Key Laboratory of Coal Mine Disaster Prevention and Control, China University of Mining and Technology（中国矿业大学煤矿灾害防治控制国家重点实验室）；The Lyell Centre, Heriot-Watt University（赫瑞斯泰大学利尔中心）；Subsurface Energy Transition and Innovation Centre, Heriot-Watt University（赫瑞斯泰大学地下能源转型与创新中心）
About rectified sigmoid function for enhancing the accuracy of Physics-Informed Neural Networks
University of Nizhny Novgorod（下诺夫哥罗德国立大学）；Huawei Nizhny Novgorod Research Center（华为下诺夫哥罗德研究中心）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
Are Two Hidden Layers Still Enough for the Physics-Informed Neural Networks?
Department of Radiophysics, University of Nizhny Novgorod（尼日尼诺夫戈罗德大学无线电物理系）；Huawei Nizhny Novgorod Research Center（华为尼日尼诺夫戈罗德研究中心）；Skolkovo Institute of Science and Technology（斯克尔科沃科学技术研究院）
Physics-informed neural networks (PINNs) have recently received much attention due to their capabilities in solving both forward and inverse problems. For training a deep neural network associated with a PINN, one typically constructs a total loss function using a weighted sum of different loss terms and then tries to minimize that. This approach often becomes problematic for solving stiff equations since it cannot consider adaptive increments. Many studies reported the poor performance of the PINN and its challenges in simulating stiff chemical active issues with administering conditions of s…
The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks
BMW AG, Digital Campus Munich（宝马集团慕尼黑数字园区）；Basque Center for Applied Mathematics（巴斯克应用数学中心）；University of the Basque Country（巴斯克大学）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Munich Data Science Institute（慕尼黑数据科学研究所）
Pacific Northwest National Laboratory（太平洋西北国家实验室）；INTERA Incorporated（INTERA公司）
Data assimilation for parameter and state estimation in subsurface transport problems remains a significant challenge due to the sparsity of measurements, the heterogeneity of porous media, and the high computational cost of forward numerical models. We present a physics-informed deep neural networks (DNNs) machine learning method for estimating space-dependent hydraulic conductivity, hydraulic head, and concentration fields from sparse measurements. In this approach, we employ individual DNNs to approximate the unknown parameters (e.g., hydraulic conductivity) and states (e.g., hydraulic head…
