一种计算深度前馈神经网络高阶导数的拟线性算法
A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks
AI总结：
针对神经网络求解微分方程时高阶导数自动微分耗时指数增长的问题，提出$n$-TangentProp拟线性算法，可高效计算全连接前馈网络高阶导数，在物理信息神经网络中训练速度提升显著且扩展性良好。
摘要：
由于计算高阶导数时自动微分的运行时间呈指数级增长，使用神经网络求解微分方程在实践中存在困难。我们提出$n$-TangentProp，它是TangentProp（切向传播）形式体系\footnote{simard1991tangent}向任意多阶导数的自然扩展。对于具有光滑无参数激活函数的全连接前馈神经网络$f$，$n$-TangentProp能以拟线性时间而非指数时间计算精确导数$d^n/dx^n f(x)$。我们在不同深度、宽度和导数阶数范围内通过实验验证了该算法。我们证明该方法在物理信息神经网络（physics-informed neural networks）场景中尤其有益，其中$n$-TangentProp（简称\ntp）的训练速度比以往方法快得多，并且在模型规模和以所需导数数量衡量的损失函数复杂度方面都具有良好的扩展性。本文代码可在https://github.com/kyrochi/n_tangentprop 获取。
英文摘要：
The use of neural networks for solving differential equations is practically difficult due to the exponentially increasing runtime of autodifferentiation when computing high-order derivatives. We propose $n$-TangentProp, the natural extension of the TangentProp formalism \cite{simard1991tangent} to arbitrarily many derivatives. $n$-TangentProp computes the exact derivative $d^n/dx^n f(x)$ in quasilinear, instead of exponential time, for a densely connected, feed-forward neural network $f$ with a smooth, parameter-free activation function. We validate our algorithm empirically across a range of depths, widths, and number of derivatives. We demonstrate that our method is particularly beneficial in the context of physics-informed neural networks where \ntp allows for significantly faster training times than previous methods and has favorable scaling with respect to both model size and loss-function complexity as measured by the number of required derivatives. The code for this paper can be found at https://github.com/kyrochi/n\_tangentprop.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
