Neural networks can be used to learn the solution of partial differential equations (PDEs) on arbitrary domains without requiring a computational mesh. Common approaches integrate differential operators in training neural networks using a structured loss function. The most common training algorithm for neural networks is backpropagation which relies on the gradient of the loss function with respect to the parameters of the network. In this work, we characterize the difficulty of training neural networks on physics by investigating the impact of differential operators in corrupting the back propagated gradients. Particularly, we show that perturbations present in the output of a neural network model during early stages of training lead to higher levels of noise in a structured loss function that is composed of high-order differential operators. These perturbations consequently corrupt the back-propagated gradients and impede convergence. We mitigate this issue by introducing auxiliary flux parameters to obtain a system of first-order differential equations. We formulate a non-linear unconstrained optimization problem using the augmented Lagrangian method that properly constrains the boundary conditions and adaptively focus on regions of higher gradients that are difficult to learn. We apply our approach to learn the solution of various benchmark PDE problems and demonstrate orders of magnitude improvement over existing approaches.
The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning
School of Computer Software, Tianjin University（天津大学计算机软件学院）；International Joint Institute of Tianjin University, Fuzhou（天津大学福州国际联合学院）；The Hong Kong University of Science and Technology（香港科技大学）
A Markovian View of Iterative-Feedback Loops in Image Generative Models: Neural Resonance and Model Collapse
Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, USA（印第安纳大学信息学、计算与工程学院）；Department of Computer Science, Stonehill College, Easton, MA, USA（石溪学院计算机科学系）
Objective Value Change and Shape-Based Accelerated Optimization for the Neural Network Approximation
Applied Mathematics and Computational Research Division, Lawrence Berkeley National Laboratory（应用数学与计算研究部，劳伦斯伯克利国家实验室）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学学院，西安交通大学）；School of Mathematics, Southwest Jiaotong University（数学学院，西南交通大学）
