arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.02192cs.LGcs.NAmath.NA
收藏

神经ODE训练中自动微分的修正

Correcting Auto-Differentiation in Neural-ODE Training

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Massachusetts Institute of Technology(麻省理工学院)

Yewei Xu, Shi Chen, Qin Li

AI总结:

研究探讨了在神经ODE架构中使用自动微分对深度神经网络更新的影响,发现高阶方法可能导致梯度震荡,提出后处理技术以消除震荡并提升收敛性。

摘要:

自动微分在深度神经网络(DNN)训练中是否能提供合理的更新?当DNN遵循神经ODE架构时,能否信任自动微分提供的梯度?通过数学分析和数值证据表明,当神经网络使用高阶方法如线性多步法(LMM)或显式龙格-库塔法(ERK)近似底层ODE流时,暴力自动微分常引入人工振荡,阻碍收敛。在Leapfrog和二阶段ERK情况下,提出简单的后处理技术以消除振荡,修正梯度计算并恢复准确更新。

英文摘要:

Does the use of auto-differentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by auto-differentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as Linear Multistep Methods (LMM) or Explicit Runge-Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force auto-differentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of Leapfrog and 2-stage ERK, we propose simple post-processing techniques that effectively eliminates these oscillations, correct the gradient computation and thus returns the accurate updates.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑