arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1808.00079cs.LGcs.DCstat.ML
收藏

任意计算图的最优梯度检查点搜索

Optimal Gradient Checkpoint Search for Arbitrary Computation Graphs

Carnegie Mellon University(卡内基梅隆大学)

Jianwei Feng, Dong Huang

摘要:

深度神经网络(DNN)在现代图像/视频数据库上训练时需要巨大的GPU内存。不幸的是,GPU内存在物理上是有限的,这限制了图像分辨率和批处理大小,这些图像分辨率和批处理大小可以用于训练以获得更好的DNN性能。与需要物理升级GPU的解决方案不同,梯度检查点(GCP)训练将计算转换为超出现有GPU硬件的更多内存。GCP只存储中间张量的子集,称为梯度检查点(GC),在向前。然后在向后期间,进行额外的局部向前以计算丢失的张量。总的训练内存成本变为(1)梯度检查点的内存成本和(2)本地转发的最大内存成本之和。为了实现最大的内存截止,需要最优算法来选择GC。现有的GCP方法要么依赖于GC的手动输入,要么依赖于线性计算图(LCG)上基于几何学的GC搜索,并且不能应用于任意计算图(ACG)。在本文中,我们提出的理论和优化算法GC选择,第一次,适用于ACG和实现最大的内存截止。大量的实验表明,我们的方法不仅优于现有的方法(仅适用于LCG),并适用于一个庞大的家庭的LCG和ACG网络,如Alexnet,VGG,ResNet,Densenet,Inception Net和高度复杂的DNN通过网络架构搜索。我们的工作使ACG上的GCP训练成为可能,并以适度的时间开销(约30%-50%)切断高达80%的训练记忆。码可用

英文摘要:

Deep Neural Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory is physically finite, which limits the image resolutions and batch sizes that could be used in training for better DNN performance. Unlike solutions that require physically upgrade GPUs, the Gradient CheckPointing(GCP) training trades computation for more memory beyond existing GPU hardware. GCP only stores a subset of intermediate tensors, called Gradient Checkpoints (GCs), during forward. Then during backward, extra local forwards are conducted to compute the missing tensors. The total training memory cost becomes the sum of (1) the memory cost of the gradient checkpoints and (2) the maximum memory cost of local forwards. To achieve maximal memory cut-offs, one needs optimal algorithms to select GCs. Existing GCP approaches rely on either manual input of GCs or heuristics-based GC search on Linear Computation Graphs (LCGs), and cannot apply to Arbitrary Computation Graphs(ACGs). In this paper, we present theories and optimal algorithms on GC selection that, for the first time, are applicable to ACGs and achieve the maximal memory cut-offs. Extensive experiments show that our approach not only outperforms existing approaches (only applicable on LCGs), and is applicable to a vast family of LCG and ACG networks, such as Alexnet, VGG, ResNet, Densenet, Inception Net and highly complicated DNNs by Network Architecture Search. Our work enables GCP training on ACGs, and cuts off up-to 80% of training memory with a moderate time overhead (~30%-50%). Codes are available

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑