arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.09689stat.MLcs.LG
收藏

AntisymmetricRNN:循环神经网络的动态系统观

AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

University of British Columbia(英属哥伦比亚大学);Google Brain(谷歌大脑)

Bo Chang, Minmin Chen, Eldad Haber, Ed H. Chi

摘要:

循环神经网络在序列数据建模中得到了广泛的应用。然而,由于梯度的爆炸或消失,使用这些模型学习长期依赖关系仍然很困难。在本文中,我们绘制了递归网络和常微分方程之间的联系。在这个理论框架下,提出了一种特殊形式的递归网络,称为Anti-mixicRNN,由于其基础微分方程的稳定性,它能够捕获长期依赖性。现有的提高RNN可训练性的方法通常会产生显著的计算开销。相比之下,AnticaRNN通过设计实现了相同的目标。我们展示了这种新架构的优势,通过广泛的模拟和实验。AntimarticRNN表现出更可预测的动态。它在需要长期记忆的任务上优于常规LSTM模型,并且与短期依赖占主导地位的任务的性能相匹配,尽管它要简单得多。

英文摘要:

Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑