Does the use of auto-differentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by auto-differentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as Linear Multistep Methods (LMM) or Explicit Runge-Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force auto-differentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of Leapfrog and 2-stage ERK, we propose simple post-processing techniques that effectively eliminates these oscillations, correct the gradient computation and thus returns the accurate updates.
A physics-informed neural network method for the approximation of slow invariant manifolds for the general class of stiff systems of ODEs
Modelling Engineering Risk and Complexity, Scuola Superiore Meridionale(工程风险与复杂性建模,南方高级学校);Institute of Science and Technology for Energy and Sustainable Mobility, Consiglio Nazionale delle Ricerche(能源与可持续移动科技研究所,国家研究理事会)
Interpretable Neural ODEs for Gene Regulatory Network Discovery under Perturbations
Department of Computer Science, Columbia University, New York, U.S.(哥伦比亚大学计算机科学系);Department of Industrial Engineering and Operations Research, Columbia University, New York, U.S.(哥伦比亚大学工业工程与运筹学系);Department of Applied Mathematics and Applied Physics, Columbia University, New York, U.S.(哥伦比亚大学应用数学与应用物理系);New York Genome Center, New York, U.S.(纽约基因组中心);Irving Institute of Cancer Dynamics, New York, U.S.(伊万·罗伯特癌症动力学研究所);Institute of Computational Biology, Helmholtz Munich, Munich, Germany(海德堡医学院计算生物学研究所);Department of Mathematics, Technische Universität München, Munich, Germany(慕尼黑技术大学数学系)
Physics-Informed Neural Networks: A Didactic Derivation of the Complete Training Cycle
Campus de Engenharias e Ciências Agrárias (CECA), Universidade Federal de Alagoas (UFAL), Rio Largo, Alagoas, 57100-000, Brazil(工程与农业科学学院(CECA),阿拉加斯联邦大学(UFAL),里奥拉戈,阿拉加斯,57100-000,巴西)
Identification of Dynamical Systems using Symbolic Regression
Josef Ressel Centre for Symbolic Regression(约瑟夫·雷塞尔符号回归中心);Heuristic and Evolutionary Algorithms Laboratory(启发式与进化算法实验室);University of Applied Sciences Upper Austria(上奥地利应用科学大学)
Department of Computational Science and Philosophy(计算科学与哲学系);Frankfurt School of Finance and Management(法兰克福金融与管理学院);Laboratory for Systems Medicine(系统医学实验室);University of Florida(佛罗里达大学)
Comparing Physics-Informed and Neural ODE Approaches for Modeling Nonlinear Biological Systems: A Case Study Based on the Morris-Lecar Model
Department of Education, School of Pedagogical and Technological Education (ASPETE)(教育与技术教育学院(ASPETE)教育系);Department of Mechanical Engineering Educators, School of Pedagogical and Technological Education (ASPETE)(教育与技术教育学院(ASPETE)机械工程教育系)
Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics
Indian Institute of Technology Kanpur(印度理工学院坎普尔分校);Julia Computing;Massachusetts Institute of Technology(麻省理工学院);University of Maryland Baltimore(马里兰大学巴尔的摩分校);Pumas AI