arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.09457cs.LGmath.CA
收藏

“Hey,that's not an ODE“:通过Seminorms实现更快的ODE伴随

"Hey, that's not an ODE": Faster ODE Adjoints via Seminorms

University of Oxford(牛津大学);The Alan Turing Institute(艾伦·图灵研究所);University of Toronto(多伦多大学);The Vector Institute(向量研究所)

Patrick Kidger, Ricky T. Q. Chen, Terry Lyons

摘要:

神经微分方程可以经由伴随方法通过反向传播梯度来训练,伴随方法是通常使用自适应步长数值微分方程求解器来求解的另一微分方程。如果所建议的步骤的误差\emph{relative to some norm}足够小,则接受所建议的步骤;否则拒绝所建议的步骤,缩小步骤,并重复该过程。在这里,我们证明了伴随方程的特殊结构使得通常的范数选择(如$L^2$)不必要的严格。通过用更合适的(半)范数替换它,不必要地拒绝更少的步骤,并且使反向传播更快。这只需要对代码进行一些小的修改。在广泛的任务上的实验-包括时间序列,生成建模和物理控制-证明了减少40%的功能评估的中值改进。在某些问题上,我们看到函数求值减少了62%,因此总体训练时间大约减少了一半。

英文摘要:

Neural differential equations may be trained by backpropagating gradients via the adjoint method, which is another differential equation typically solved using an adaptive-step-size numerical differential equation solver. A proposed step is accepted if its error, \emph{relative to some norm}, is sufficiently small; else it is rejected, the step is shrunk, and the process is repeated. Here, we demonstrate that the particular structure of the adjoint equations makes the usual choices of norm (such as $L^2$) unnecessarily stringent. By replacing it with a more appropriate (semi)norm, fewer steps are unnecessarily rejected and the backpropagation is made faster. This requires only minor code modifications. Experiments on a wide range of tasks -- including time series, generative modeling, and physical control -- demonstrate a median improvement of 40% fewer function evaluations. On some problems we see as much as 62% fewer function evaluations, so that the overall training time is roughly halved.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑