arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.14501cs.LGcs.CV
收藏

深度网络的内存优化

Memory Optimization for Deep Networks

University of Texas at Austin(德克萨斯大学奥斯汀分校);VMware Research(VMware研究院)

Aashaka Shah, Chao-Yuan Wu, Jayashree Mohan, Vijay Chidambaram, Philipp Krähenbühl

摘要:

深度学习正在缓慢但稳步地遇到内存瓶颈。虽然顶级GPU中的张量计算在过去五年中增长了32倍,但总可用内存仅增长了2.5倍。这阻止了研究人员探索更大的架构,因为训练大型网络需要更多的内存来存储中间输出。在本文中,我们提出了MONeT,这是一个自动框架,可以最大限度地减少深度网络的内存占用和计算开销。MONetT共同优化检查点调度和各种运营商的实现。MONeT能够超越所有以前的手动调优操作以及自动检查点。MONeT将各种PyTorch模型的总体内存需求降低了3倍,计算开销为9-16%。对于相同的计算成本,MONeT需要的内存比当前最先进的自动检查点框架少1.2- 1.8倍。我们的代码可在https://github.com/utsaslab/MONeT上获得。

英文摘要:

Deep learning is slowly, but steadily, hitting a memory bottleneck. While the tensor computation in top-of-the-line GPUs increased by 32x over the last five years, the total available memory only grew by 2.5x. This prevents researchers from exploring larger architectures, as training large networks requires more memory for storing intermediate outputs. In this paper, we present MONeT, an automatic framework that minimizes both the memory footprint and computational overhead of deep networks. MONeT jointly optimizes the checkpointing schedule and the implementation of various operators. MONeT is able to outperform all prior hand-tuned operations as well as automated checkpointing. MONeT reduces the overall memory requirement by 3x for various PyTorch models, with a 9-16% overhead in computation. For the same computation cost, MONeT requires 1.2-1.8x less memory than current state-of-the-art automated checkpointing frameworks. Our code is available at https://github.com/utsaslab/MONeT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑