By integrating domain knowledge with labeled samples, informed machine learning has been emerging to improve the learning performance for a wide range of applications. Nonetheless, rigorous understanding of the role of injected domain knowledge has been under-explored. In this paper, we consider an informed deep neural network (DNN) with over-parameterization and domain knowledge integrated into its training objective function, and study how and why domain knowledge benefits the performance. Concretely, we quantitatively demonstrate the two benefits of domain knowledge in informed learning - regularizing the label-based supervision and supplementing the labeled samples - and reveal the trade-off between label and knowledge imperfectness in the bound of the population risk. Based on the theoretical analysis, we propose a generalized informed training objective to better exploit the benefits of knowledge and balance the label and knowledge imperfectness, which is validated by the population risk bound. Our analysis on sampling complexity sheds lights on how to choose the hyper-parameters for informed learning, and further justifies the advantages of knowledge informed learning.
Efficient Global-Local Fusion Sampling for Physics-Informed Neural Networks
School of Mathematical Sciences, Soochow University（苏州大学数学科学学院）；Zu Chongzhi Center , Duke Kunshan University（杜克昆山学院钟楚中心）；University of Science and Technology of China（中国科学技术大学）；Key Laboratory of the Ministry of Education for Mathematical Foundations and Applications of Digital Technology（教育部数学基础与应用数字技术重点实验室）
University of Pennsylvania（宾夕法尼亚大学）；McGill University（麦吉尔大学）
This paper presents a distributed scalable multi-robot planning algorithm for informed sampling of quasistatic spatial fields. We address the problem of efficient data collection using multiple autonomous vehicles and consider the effects of communication between multiple robots, acting independently, on the overall sampling performance of the team. We focus on the distributed sampling problem where the robots operate independent of their teammates, but have the ability to communicate their current state to other neighbors within a fixed communication range. Our proposed approach is scalable a…
Gradient-Informed Temporal Sampling Improves Rollout Accuracy in PDE Surrogate Training
School of Future Technology, South China University of Technology, China（未来技术学院，华南理工大学，中国）；State Key Laboratory of Ocean Sensing & Ocean College, Zhejiang University, China（海洋感知国家重点实验室及海洋学院，浙江大学，中国）；Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology, USA（天体物理与空间研究所，麻省理工学院，美国）
Physics-Informed Neural Networks (PINNs) have become a kind of attractive machine learning method for obtaining solutions of partial differential equations (PDEs). Training PINNs can be seen as a semi-supervised learning task, in which only exact values of initial and boundary points can be obtained in solving forward problems, and in the whole spatio-temporal domain collocation points are sampled without exact labels, which brings training difficulties. Thus the selection of collocation points and sampling methods are quite crucial in training PINNs. Existing sampling methods include fixed an…
An Imbalanced Learning-based Sampling Method for Physics-informed Neural Networks
The Fields Institute for Research in Mathematical Sciences（数学科学菲尔兹研究所）；Pennsylvania State University（宾夕法尼亚州立大学）；Data Science Research Center, Duke Kunshan University（昆山杜克大学数据科学研究中心）；Zu Chongzhi Center for Mathematics and Computational Sciences, Duke Kunshan University（昆山杜克大学祖冲之数学与计算科学中心）
Operator-informed initialization for Fourier features physics-informed neural networks
National Center for Artificial Intelligence（国家人工智能中心）；University of Pennsylvania（宾夕法尼亚大学）；Pontificia Universidad Católica de Chile（智利天主教大学）；Millennium Institute for Intelligent Healthcare Engineering, iHEALTH（千禧智能医疗工程研究所）
Deep-learning models for language generation tasks tend to produce repetitive output. Various methods have been proposed to encourage lexical diversity during decoding, but this often comes at a cost to the perceived fluency and adequacy of the output. In this work, we propose to ameliorate this cost by using an Imitation Learning approach to explore the level of diversity that a language generation model can reliably produce. Specifically, we augment the decoding process with a meta-classifier trained to distinguish which words at any given timestep will lead to high-quality output. We focus …
Accelerating 4D Hyperspectral Imaging through Physics-Informed Neural Representation and Adaptive Sampling
Dept. of Electrical and Computer Engineering, UC San Diego（加州大学圣地亚哥分校电气与计算机工程系）；Dept. of Chemistry, UC San Diego（加州大学圣地亚哥分校化学系）；Materials Science and Engineering Program, UC San Diego（加州大学圣地亚哥分校材料科学与工程项目）
In this work, we propose a new Gaussian process regression (GPR)-based multifidelity method: physics-informed CoKriging (CoPhIK). In CoKriging-based multifidelity methods, the quantities of interest are modeled as linear combinations of multiple parameterized stationary Gaussian processes (GPs), and the hyperparameters of these GPs are estimated from data via optimization. In CoPhIK, we construct a GP representing low-fidelity data using physics-informed Kriging (PhIK), and model the discrepancy between low- and high-fidelity data using a parameterized GP with hyperparameters identified via op…
Evaluation of Sampling Strategies and Physics-Informed Kolmogorov--Arnold Networks in Unbounded Domains
Mathematical Applications in Science and Engineering, School of Applied Sciences and Engineering, Universidad EAFIT, Medellín, Colombia（科学与工程数学应用，应用科学与工程学院，埃法伊特大学，梅德林，哥伦比亚）；Nature and City, School of Applied Sciences and Engineering, Universidad EAFIT, Medellín, Colombia（自然与城市，应用科学与工程学院，埃法伊特大学，梅德林，哥伦比亚）
Sampling-based Motion Planners (SMPs) have become increasingly popular as they provide collision-free path solutions regardless of obstacle geometry in a given environment. However, their computational complexity increases significantly with the dimensionality of the motion planning problem. Adaptive sampling is one of the ways to speed up SMPs by sampling a particular region of a configuration space that is more likely to contain an optimal path solution. Although there are a wide variety of algorithms for adaptive sampling, they rely on hand-crafted heuristics; furthermore, their performance…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；NVIDIA（英伟达）
Physics-Informed Neural Networks (PINNs) are a class of deep neural networks that are trained, using automatic differentiation, to compute the response of systems governed by partial differential equations (PDEs). The training of PINNs is simulation-free, and does not require any training dataset to be obtained from numerical PDE solvers. Instead, it only requires the physical problem description, including the governing laws of physics, domain geometry, initial/boundary conditions, and the material properties. This training usually involves solving a non-convex optimization problem using vari…
Energy Dissipation Rate Guided Adaptive Sampling for Physics-Informed Neural Networks: Resolving Surface-Bulk Dynamics in Allen-Cahn Systems
Department of Mathematics, The Pennsylvania State University（数学系，宾夕法尼亚州立大学）；Department of Mathematics, The Hong Kong University of Science and Technology（数学系，香港科学与技术大学）；Department of Mathematics, University of South Carolina（数学系，南卡罗来纳大学）
Defense Innovation Institute, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）；Human Resource Center（人力资源中心）
Learning solutions of partial differential equations (PDEs) with Physics-Informed Neural Networks (PINNs) is an attractive alternative approach to traditional solvers due to its flexibility and ease of incorporating observed data. Despite the success of PINNs in accurately solving a wide variety of PDEs, the method still requires improvements in terms of computational efficiency. One possible improvement idea is to optimize the generation of training point sets. Residual-based adaptive sampling and quasi-uniform sampling approaches have been each applied to improve the training effects of PINN…
Virginia Tech（弗吉尼亚理工大学）；University of Michigan（密歇根大学）
Physics-informed Neural Networks (PINNs) are gaining attention in the engineering and scientific literature for solving a range of differential equations with applications in weather modeling, healthcare, manufacturing, etc. Poor scalability is one of the barriers to utilizing PINNs for many real-world problems. To address this, a Self-scalable tanh (Stan) activation function is proposed for the PINNs. The proposed Stan function is smooth, non-saturating, and has a trainable parameter. During training, it can allow easy flow of gradients to compute the required derivatives and also enable syst…
Physics informed neural networks (PINNs) are deep learning based techniques for solving partial differential equations (PDEs) encounted in computational science and engineering. Guided by data and physical laws, PINNs find a neural network that approximates the solution to a system of PDEs. Such a neural network is obtained by minimizing a loss function in which any prior knowledge of PDEs and data are encoded. Despite its remarkable empirical success in one, two or three dimensional problems, there is little theoretical justification for PINNs. As the number of data grows, PINNs generate a se…
SIREM: Speech-Informed MRI Reconstruction with Learned Sampling
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃森哲-埃尔朗根-纽伦堡大学模式识别实验室）；Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根大学医院放射学研究所）；Institut für Informationsverarbeitung, Leibniz Universität Hannover（汉诺威莱比锡大学信息处理研究所）；Department of Radiology, Harvard Medical School and Massachusetts General Hospital（哈佛医学院放射科和麻省总医院）
National Innovation Institute of Defense Technology, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）
Physics-Informed Neural Networks (PINNs) can be regarded as general-purpose PDE solvers, but it might be slow to train PINNs on particular problems, and there is no theoretical guarantee of corresponding error bounds. In this manuscript, we propose a variant called Prior Dictionary based Physics-Informed Neural Networks (PD-PINNs). Equipped with task-dependent dictionaries, PD-PINNs enjoy enhanced representation power on the tasks, which helps to capture features provided by dictionaries so that the proposed neural networks can achieve faster convergence in the process of training. In various …
Using vanilla NeuralODEs to model large and/or complex systems often fails due two reasons: Stability and convergence. NeuralODEs are capable of describing stable as well as instable dynamic systems. Selecting an appropriate numerical solver is not trivial, because NeuralODE properties change during training. If the NeuralODE becomes more stiff, a suboptimal solver may need to perform very small solver steps, which significantly slows down the training process. If the NeuralODE becomes to instable, the numerical solver might not be able to solve it at all, which causes the training process to …
Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）；Robotics and Big Data Laboratory, University of Haifa（海法大学机器人与大数据实验室）
We introduce a pruning algorithm that provably sparsifies the parameters of a trained model in a way that approximately preserves the model's predictive accuracy. Our algorithm uses a small batch of input points to construct a data-informed importance sampling distribution over the network's parameters, and adaptively mixes a sampling-based and deterministic pruning procedure to discard redundant weights. Our pruning method is simultaneously computationally efficient, provably accurate, and broadly applicable to various network architectures and data distributions. Our empirical comparisons sh…
Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs). However, they often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under stochastic gradient descent with momentum (SGDM). This demonstrates SGDM significantly reduces the effect of spectral bias. We have also examined why …
Convergence of physics-informed neural networks modeling time-harmonic wave fields
Institute of Fundamentals and Theory of Electrical Engineering（电气工程基础与理论研究所）；Graz University of Technology（格拉茨技术大学）；Faculty of Electrical Engineering（电气工程学院）；Czech Technical University in Prague（布拉格捷克技术大学）
An important inference from Neural Tangent Kernel (NTK) theory is the existence of spectral bias (SB), that is, low frequency components of the target function of a fully connected Artificial Neural Network (ANN) being learnt significantly faster than the higher frequencies during training. This is established for Mean Square Error (MSE) loss functions with very low learning rate parameters. Physics Informed Neural Networks (PINNs) are designed to learn the solutions of differential equations (DE) of arbitrary orders; in PINNs the loss functions are obtained as the residues of the conservative…
