Deep Neural Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory is physically finite, which limits the image resolutions and batch sizes that could be used in training for better DNN performance. Unlike solutions that require physically upgrade GPUs, the Gradient CheckPointing(GCP) training trades computation for more memory beyond existing GPU hardware. GCP only stores a subset of intermediate tensors, called Gradient Checkpoints (GCs), during forward. Then during backward, extra local forwards are conducted to compute the missing tensors. The total training memory cost becomes the sum of (1) the memory cost of the gradient checkpoints and (2) the maximum memory cost of local forwards. To achieve maximal memory cut-offs, one needs optimal algorithms to select GCs. Existing GCP approaches rely on either manual input of GCs or heuristics-based GC search on Linear Computation Graphs (LCGs), and cannot apply to Arbitrary Computation Graphs(ACGs). In this paper, we present theories and optimal algorithms on GC selection that, for the first time, are applicable to ACGs and achieve the maximal memory cut-offs. Extensive experiments show that our approach not only outperforms existing approaches (only applicable on LCGs), and is applicable to a vast family of LCG and ACG networks, such as Alexnet, VGG, ResNet, Densenet, Inception Net and highly complicated DNNs by Network Architecture Search. Our work enables GCP training on ACGs, and cuts off up-to 80% of training memory with a moderate time overhead (~30%-50%). Codes are available
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
GPU Memory Usage Optimization for Backward Propagation in Deep Network Training
Institute of Information Science, Academia Sinica(中央研究院资讯科学研究所);Department of Computer Science and Information Engineering, National Taiwan University(台湾大学资讯工程学系)
View Space: Learning Representation across Arbitrary Graphs
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea(韩国釜山科学技术院电子工程学院);Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea(首尔国立大学计算机科学与工程系)
Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining
Beijing University of Civil Engineering and Architecture(北京建筑大学);Beijing Institute of Mathematical Sciences and Applications (BIMSA)(北京雁栖湖应用数学研究院);Institute of Statistics and Big Data, Renmin University of China(中国人民大学统计与大数据研究院)