延迟微分神经网络
Delay Differential Neural Networks
摘要:
神经常微分方程(NODE)将中间特征向量的计算视为由神经网络参数化的常微分方程的轨迹。在本文中,我们提出了一种新的模型,延迟微分神经网络(DDNN),延迟微分方程(DDE)的启发。该模型将隐藏特征向量的导数视为当前特征向量和过去特征向量(历史)的函数。该函数被建模为神经网络,因此,它导致了许多最近ResNet变体的连续深度替代方案。我们提出了两种不同的DDNN架构,这取决于当前和过去的特征向量被认为是。为了训练DDNN,我们提供了一种内存高效的伴随方法来计算梯度并通过网络反向传播。DDNN在不影响泛化性能的前提下,通过进一步减少参数数量,提高了NODE的数据效率。在Cifar 10和Cifar 100等合成和真实图像分类数据集上进行的实验表明了所提模型的有效性。
英文摘要:
Neural ordinary differential equations (NODEs) treat computation of intermediate feature vectors as trajectories of ordinary differential equation parameterized by a neural network. In this paper, we propose a novel model, delay differential neural networks (DDNN), inspired by delay differential equations (DDEs). The proposed model considers the derivative of the hidden feature vector as a function of the current feature vector and past feature vectors (history). The function is modelled as a neural network and consequently, it leads to continuous depth alternatives to many recent ResNet variants. We propose two different DDNN architectures, depending on the way current and past feature vectors are considered. For training DDNNs, we provide a memory-efficient adjoint method for computing gradients and back-propagate through the network. DDNN improves the data efficiency of NODE by further reducing the number of parameters without affecting the generalization performance. Experiments conducted on synthetic and real-world image classification datasets such as Cifar10 and Cifar100 show the effectiveness of the proposed models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。