MALI：一种记忆高效的神经常微分方程逆精度积分器
MALI: A memory efficient and reverse accurate integrator for Neural ODEs
摘要：
神经常微分方程（Neural ODE）是一类新的具有连续深度的深度学习模型。然而，在连续情况下的梯度的数值估计没有得到很好的解决：伴随方法的现有实现遭受逆时间轨迹的不准确性，而朴素方法和自适应检查点伴随方法（ACA）具有随积分时间增长的存储器成本。在这个项目中，基于异步蛙跳（ALF）求解器，我们提出了内存有效的ALF积分器（MALI），它具有恒定的内存成本\textit{w.r.t}数量的求解器步骤的集成类似于伴随方法，并保证在逆时间轨迹的精度（因此在梯度估计的精度）。我们在各种任务中验证了MALI：在图像识别任务中，据我们所知，MALI是第一个在ImageNet上实现神经ODE可行训练的方法，并且优于经过良好调整的ResNet，而现有方法由于内存负担重或不准确而失败;对于时间序列建模，MALI显着优于伴随方法;对于连续生成模型，MALI实现了新的最先进性能。我们在\url{https://jzkay12.github.io/TorchDiffEqPack/}提供pypi包
英文摘要：
Neural ordinary differential equations (Neural ODEs) are a new family of deep-learning models with continuous depth. However, the numerical estimation of the gradient in the continuous case is not well solved: existing implementations of the adjoint method suffer from inaccuracy in reverse-time trajectory, while the naive method and the adaptive checkpoint adjoint method (ACA) have a memory cost that grows with integration time. In this project, based on the asynchronous leapfrog (ALF) solver, we propose the Memory-efficient ALF Integrator (MALI), which has a constant memory cost \textit{w.r.t} number of solver steps in integration similar to the adjoint method, and guarantees accuracy in reverse-time trajectory (hence accuracy in gradient estimation). We validate MALI in various tasks: on image recognition tasks, to our knowledge, MALI is the first to enable feasible training of a Neural ODE on ImageNet and outperform a well-tuned ResNet, while existing methods fail due to either heavy memory burden or inaccuracy; for time series modeling, MALI significantly outperforms the adjoint method; and for continuous generative models, MALI achieves new state-of-the-art performance. We provide a pypi package at \url{https://jzkay12.github.io/TorchDiffEqPack/}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
