arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.02493stat.MLcs.LG
收藏

神经元ODE梯度估计的自适应检查点伴随方法

Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE

Yale University(耶鲁大学);Yale School of Medicine(耶鲁医学院)

Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, James Duncan

摘要:

神经常微分方程(NODE)最近引起了越来越多的关注;然而,它们在基准任务(例如图像分类)上的经验性能明显劣于离散层模型。我们证明了其性能较差的解释是现有梯度估计方法的不准确性:伴随方法在反向模式积分中具有数值误差;朴素方法直接通过ODE求解器反向传播,但在搜索最佳步长时会遇到冗余的深度计算图。我们提出了自适应检查点伴随(ACA)方法:在自动微分中,ACA应用轨迹检查点策略,将正向模式轨迹记录为反向模式轨迹以保证精度; ACA删除浅层计算图的冗余组件; ACA支持自适应求解器。在图像分类任务中,与伴随方法和朴素方法相比,ACA在一半的训练时间内实现了一半的错误率;用ACA训练的NODE在准确率和重测信度方面都优于ResNet。在时间序列建模方面,ACA优于竞争方法。最后,在一个例子中的三体问题,我们表明节点与ACA可以结合物理知识,以达到更好的精度。我们提供了ACA的PyTorch实现:\url{https://github.com/juntang-zhuang/torch-ACA}。

英文摘要:

Neural ordinary differential equations (NODEs) have recently attracted increasing attention; however, their empirical performance on benchmark tasks (e.g. image classification) are significantly inferior to discrete-layer models. We demonstrate an explanation for their poorer performance is the inaccuracy of existing gradient estimation methods: the adjoint method has numerical errors in reverse-mode integration; the naive method directly back-propagates through ODE solvers, but suffers from a redundantly deep computation graph when searching for the optimal stepsize. We propose the Adaptive Checkpoint Adjoint (ACA) method: in automatic differentiation, ACA applies a trajectory checkpoint strategy which records the forward-mode trajectory as the reverse-mode trajectory to guarantee accuracy; ACA deletes redundant components for shallow computation graphs; and ACA supports adaptive solvers. On image classification tasks, compared with the adjoint and naive method, ACA achieves half the error rate in half the training time; NODE trained with ACA outperforms ResNet in both accuracy and test-retest reliability. On time-series modeling, ACA outperforms competing methods. Finally, in an example of the three-body problem, we show NODE with ACA can incorporate physical knowledge to achieve better accuracy. We provide the PyTorch implementation of ACA: \url{https://github.com/juntang-zhuang/torch-ACA}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑