arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1702.07805cs.NE
收藏

NARX循环神经网络的分析与开发

Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies

Robert DiPietro, Christian Rupprecht, Nassir Navab, Gregory D. Hager

摘要:

循环神经网络(RNN)在许多不同的任务上都取得了最先进的性能,从机器翻译到手术活动识别,但训练RNN捕获长期依赖性仍然很困难。到目前为止,绝大多数成功的RNN架构使用状态之间的近似加法连接来缓解这个问题,如长短期记忆(LSTM)所引入的。我们采用正交方法并引入MIST RNN,这是一种NARX RNN架构,允许从遥远的过去进行直接连接。我们发现,MIST RNN 1)与LSTM和以前提出的NARX RNN相比,具有更好的消失梯度特性; 2)比以前提出的NARX RNN架构更有效,需要的计算比LSTM更少; 3)在需要非常长期依赖关系的任务上,与LSTM和Clockwork RNN相比,性能大幅提高。

英文摘要:

Recurrent neural networks (RNNs) have achieved state-of-the-art performance on many diverse tasks, from machine translation to surgical activity recognition, yet training RNNs to capture long-term dependencies remains difficult. To date, the vast majority of successful RNN architectures alleviate this problem using nearly-additive connections between states, as introduced by long short-term memory (LSTM). We take an orthogonal approach and introduce MIST RNNs, a NARX RNN architecture that allows direct connections from the very distant past. We show that MIST RNNs 1) exhibit superior vanishing-gradient properties in comparison to LSTM and previously-proposed NARX RNNs; 2) are far more efficient than previously-proposed NARX RNN architectures, requiring even fewer computations than LSTM; and 3) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑