arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1604.06174cs.LG
收藏

使用次线性内存代价训练深度网络

Training Deep Nets with Sublinear Memory Cost

Dato. Inc(Dato公司);Massachusetts Institute of Technology(麻省理工学院)

Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin

摘要:

我们提出了一种系统的方法来减少深度神经网络训练的内存消耗。具体来说,我们设计了一个算法,成本O(sqrt(n))内存训练一个n层网络,只有一个额外的向前传递每小批的计算成本。由于许多最先进的模型都达到了GPU内存的上限,我们的算法允许探索更深、更复杂的模型,并有助于推进深度学习研究的创新。我们专注于减少训练过程中存储中间特征图和梯度的内存成本。计算图分析用于自动就地操作和内存共享优化。我们证明了用计算换取内存是可能的--用一点额外的计算成本给出一个更有效的内存训练算法。在极端的情况下,我们的分析还表明,内存消耗可以减少到O(log n),只需O(n log n)的额外成本向前计算。我们的实验表明,我们可以将1,000层深度残差网络的内存成本从48 G降低到7 G,而ImageNet问题的运行时间成本仅增加30%。类似地,在非常长的序列上训练复杂的递归神经网络时,可以观察到显着的内存成本降低。

英文摘要:

We propose a systematic approach to reduce the memory consumption of deep neural network training. Specifically, we design an algorithm that costs O(sqrt(n)) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch. As many of the state-of-the-art models hit the upper bound of the GPU memory, our algorithm allows deeper and more complex models to be explored, and helps advance the innovations in deep learning research. We focus on reducing the memory cost to store the intermediate feature maps and gradients during training. Computation graph analysis is used for automatic in-place operation and memory sharing optimizations. We show that it is possible to trade computation for memory - giving a more memory efficient training algorithm with a little extra computation cost. In the extreme case, our analysis also shows that the memory consumption can be reduced to O(log n) with as little as O(n log n) extra cost for forward computation. Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑