arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1902.07275cs.LGstat.ML
收藏

循环神经网络中记忆的理解与控制

Understanding and Controlling Memory in Recurrent Neural Networks

Technion, Israel Institute of Technology(以色列理工学院);Weizmann Institute of Science(魏茨曼科学研究所)

Doron Haviv, Alexander Rivkind, Omri Barak

摘要:

为了有效地进行顺序数据处理,需要循环神经网络(RNN)通过创建记忆来跟踪过去的事件。虽然记忆和网络的隐藏状态动态之间的关系是在过去十年中建立的,但以前在这个方向上的工作主要是描述性的,主要集中在定位感兴趣的动态对象上。特别是,目前还不清楚动态可观测量如何影响性能,它们如何形成以及它们是否可以被操纵。在这里,我们利用不同的训练协议、数据集和架构来获得一系列网络,这些网络解决了具有类似性能的延迟分类任务,同时它们外推更长延迟的能力也有很大差异。我们分析了网络的隐藏状态的动态,并揭示了这种差异的原因。每个记忆被发现与一个几乎稳定的动态状态,我们称之为“慢点”。慢点速度预测所有测试数据集、协议和架构的外推性能。此外,通过跟踪慢点的形成,我们能够了解训练方案之间差异的来源。最后,我们提出了一种新的正则化技术,是基于隐藏状态的速度和内存寿命之间的关系。我们的技术操纵这些速度,从而导致随着时间的推移记忆鲁棒性的显着改善,并可能为一类新的正则化方法铺平道路。

英文摘要:

To be effective in sequential data processing, Recurrent Neural Networks (RNNs) are required to keep track of past events by creating memories. While the relation between memories and the network's hidden state dynamics was established over the last decade, previous works in this direction were of a predominantly descriptive nature focusing mainly on locating the dynamical objects of interest. In particular, it remained unclear how dynamical observables affect the performance, how they form and whether they can be manipulated. Here, we utilize different training protocols, datasets and architectures to obtain a range of networks solving a delayed classification task with similar performance, alongside substantial differences in their ability to extrapolate for longer delays. We analyze the dynamics of the network's hidden state, and uncover the reasons for this difference. Each memory is found to be associated with a nearly steady state of the dynamics which we refer to as a 'slow point'. Slow point speeds predict extrapolation performance across all datasets, protocols and architectures tested. Furthermore, by tracking the formation of the slow points we are able to understand the origin of differences between training protocols. Finally, we propose a novel regularization technique that is based on the relation between hidden state speeds and memory longevity. Our technique manipulates these speeds, thereby leading to a dramatic improvement in memory robustness over time, and could pave the way for a new class of regularization methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑