arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.13898cs.LGcs.AIcs.NEcs.ROmath.DS
收藏

闭型连续时间神经元模型

Closed-form Continuous-time Neural Models

Massachusetts Institute of Technology(麻省理工学院);Aalborg University(奥尔堡大学);University of Vienna(维也纳大学);Institute of Science and Technology Austria(奥地利科学技术研究所)

Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, Daniela Rus

摘要:

连续时间神经过程是由微分方程(DE)建立的执行顺序决策者。然而,当它们被部署在计算机上时,它们的表达能力受到数值DE求解器的检验。这种限制大大减缓了对许多自然物理现象的扩展和理解,例如神经系统的动力学。理想情况下,我们可以通过求解封闭形式的给定动力系统来绕过这个瓶颈。这是众所周知的,是棘手的一般。在这里,我们表明,它是可能的密切近似神经元和突触之间的相互作用-天然和人工神经网络的积木-液体时间常数网络(LTC)有效地构建在封闭的形式。为此,我们计算出一个紧密有界近似的解决方案的积分出现在LTC的动力学,迄今为止还没有已知的封闭形式的解决方案。这种封闭形式的解决方案极大地影响了连续时间和连续深度神经模型的设计;例如,由于时间以封闭形式显式出现,因此该公式放松了对复杂数值求解器的需求。因此,与基于微分方程的模型相比,我们获得的模型在训练和推理方面快了一到五个数量级。更重要的是,与基于ODE的连续网络相比,与其他深度学习实例相比,封闭式网络可以非常好地扩展。最后,由于这些模型来自液体网络,与先进的递归模型相比,它们在时间序列建模方面表现出显着的性能。

英文摘要:

Continuous-time neural processes are performant sequential decision-makers that are built by differential equations (DE). However, their expressive power when they are deployed on computers is bottlenecked by numerical DE solvers. This limitation has significantly slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show it is possible to closely approximate the interaction between neurons and synapses -- the building blocks of natural and artificial neural networks -- constructed by liquid time-constant networks (LTCs) efficiently in closed-form. To this end, we compute a tightly-bounded approximation of the solution of an integral appearing in LTCs' dynamics, that has had no known closed-form solution so far. This closed-form solution substantially impacts the design of continuous-time and continuous-depth neural models; for instance, since time appears explicitly in closed-form, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared to differential equation-based counterparts. More importantly, in contrast to ODE-based continuous networks, closed-form networks can scale remarkably well compared to other deep learning instances. Lastly, as these models are derived from liquid networks, they show remarkable performance in time series modeling, compared to advanced recurrent models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑