arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1604.01946cs.LGcs.NE
收藏

基于GPU的循环神经网络性能优化

Optimizing Performance of Recurrent Neural Networks on GPUs

NVIDIA(英伟达);University of Oxford(牛津大学);Google DeepMind(谷歌DeepMind)

Jeremy Appleyard, Tomas Kocisky, Phil Blunsom

摘要:

随着循环神经网络变得越来越大、越来越深,单个网络的训练时间也会增加到几周甚至几个月。因此,有一个显着的动机,以提高这些网络的性能和可扩展性。虽然GPU已经成为训练和部署递归模型的首选硬件,但所采用的实现通常只使用这些架构的基本优化。在这篇文章中,我们演示了通过暴露网络内操作之间的并行性,可以在一个简单的实现上实现跨网络大小范围的数量级加速。我们描述了NVIDIA cuDNN第五版中包含的三个优化阶段:首先优化单个单元,其次优化单个层,第三优化整个网络。

英文摘要:

As recurrent neural networks become larger and deeper, training times for single networks are rising into weeks or even months. As such there is a significant incentive to improve the performance and scalability of these networks. While GPUs have become the hardware of choice for training and deploying recurrent models, the implementations employed often make use of only basic optimizations for these architectures. In this article we demonstrate that by exposing parallelism between operations within the network, an order of magnitude speedup across a range of network sizes can be achieved over a naive implementation. We describe three stages of optimization that have been incorporated into the fifth release of NVIDIA's cuDNN: firstly optimizing a single cell, secondly a single layer, and thirdly the entire network.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑