arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1901.06773cs.LGcs.AI
收藏

AccUDNN:一个用于训练超深神经网络的GPU内存高效加速器

AccUDNN: A GPU Memory Efficient Accelerator for Training Ultra-deep Neural Networks

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)

Jinrong Guo, Wantao Liu, Wang Wang, Qu Lu, Songlin Hu, Jizhong Han, Ruixuan Li

摘要:

通常,超深神经网络(UDNN)倾向于产生高质量的模型,但其训练过程通常是资源密集型和耗时的。现代GPU的DRAM容量不足是阻碍UDNN可训练性和训练效率的主要瓶颈。在本文中,我们提出了“AccUDNN”,一个加速器,旨在最大限度地利用有限的GPU内存资源,以加快UDNN的训练过程。AccUDNN主要包括两个模块:内存优化器和超参数调整器。内存优化器开发了一种性能模型指导的动态换出/换入策略,通过将适当的数据卸载到主机内存,可以显著削减GPU内存占用,以克服UDNN可训练性的限制。在应用内存优化策略后,设计了超参数调整器,以探索效率最优的小批量大小和匹配的学习率。评估表明,AccUDNN将ResNet-152的GPU内存需求从超过24 GB减少到8 GB。反过来,考虑到12 GB的GPU内存预算,效率最佳的minibatch大小可以达到原始Caffe的4.2倍。得益于更好地利用单个GPU的计算资源和更少的大minibatch大小的参数同步,8个GPU的集群实现了7.7倍的速度提升,没有任何通信优化,没有精度损失。

英文摘要:

Typically, Ultra-deep neural network(UDNN) tends to yield high-quality model, but its training process is usually resource intensive and time-consuming. Modern GPU's scarce DRAM capacity is the primary bottleneck that hinders the trainability and the training efficiency of UDNN. In this paper, we present "AccUDNN", an accelerator that aims to make the utmost use of finite GPU memory resources to speed up the training process of UDNN. AccUDNN mainly includes two modules: memory optimizer and hyperparameter tuner. Memory optimizer develops a performance-model guided dynamic swap out/in strategy, by offloading appropriate data to host memory, GPU memory footprint can be significantly slashed to overcome the restriction of trainability of UDNN. After applying the memory optimization strategy, hyperparameter tuner is designed to explore the efficiency-optimal minibatch size and the matched learning rate. Evaluations demonstrate that AccUDNN cuts down the GPU memory requirement of ResNet-152 from more than 24GB to 8GB. In turn, given 12GB GPU memory budget, the efficiency-optimal minibatch size can reach 4.2x larger than original Caffe. Benefiting from better utilization of single GPU's computing resources and fewer parameter synchronization of large minibatch size, 7.7x speed-up is achieved by 8 GPUs' cluster without any communication optimization and no accuracy losses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑