arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2105.14092cs.NE
收藏

最小冗余门控循环神经网络

Least Redundant Gated Recurrent Neural Network

Łukasz Neumann, Łukasz Lepak, Paweł Wawrzyński

AI总结:

本文提出深度记忆更新(DMU)循环神经网络,通过深度变换滞后状态与输入来更新记忆,训练稳定快速,实验表明其性能可媲美或超越LSTM、GRU等主流架构。

摘要:

循环神经网络是序列数据处理的重要工具,但其训练问题广为人知。挑战包括捕捉连续状态之间的复杂关系以及训练的稳定性和效率。本文提出一种名为深度记忆更新(Deep Memory Update,DMU)的循环神经架构。该架构基于利用滞后状态和网络输入的深度变换来更新先前的记忆状态。该架构能够学习使用任意非线性函数变换其内部状态。由于将学习率与模块大小相关联,其训练稳定且快速。尽管DMU基于标准组件构建,本文给出的实验结果表明,它能够与长短期记忆网络(Long Short-Term Memory)、门控循环单元(Gated Recurrent Units)和循环高速公路网络(Recurrent Highway Networks)等最先进架构竞争,并且常常超越它们。

英文摘要:

Recurrent neural networks are important tools for sequential data processing. However, they are notorious for problems regarding their training. Challenges include capturing complex relations between consecutive states and stability and efficiency of training. In this paper, we introduce a recurrent neural architecture called Deep Memory Update (DMU). It is based on updating the previous memory state with a deep transformation of the lagged state and the network input. The architecture is able to learn to transform its internal state using any nonlinear function. Its training is stable and fast due to relating its learning rate to the size of the module. Even though DMU is based on standard components, experimental results presented here confirm that it can compete with and often outperform state-of-the-art architectures such as Long Short-Term Memory, Gated Recurrent Units, and Recurrent Highway Networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑