Science and Technology on Parallel and Distributed Processing Laboratory（并行与分布处理国防科技重点实验室）；SenseTime Research（商汤科技研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
The further development of deep neural networks is hampered by the limited GPU memory resource. Therefore, the optimization of GPU memory resources is highly demanded. Swapping and recomputation are commonly applied to make better use of GPU memory in deep learning. However, as an emerging domain, several challenges remain:1)The efficiency of recomputation is limited for both static and dynamic methods. 2)Swapping requires offloading parameters manually, which incurs a great time cost. 3) There is no such dynamic and fine-grained method that involves tensor swapping together with tensor recomputation nowadays. To remedy the above issues, we propose a novel scheduler manager named DELTA(Dynamic tEnsor offLoad and recompuTAtion). To the best of our knowledge, we are the first to make a reasonable dynamic runtime scheduler on the combination of tensor swapping and tensor recomputation without user oversight. In DELTA, we propose a filter algorithm to select the optimal tensors to be released out of GPU memory and present a director algorithm to select a proper action for each of these tensors. Furthermore, prefetching and overlapping are deliberately considered to overcome the time cost caused by swapping and recomputing tensors. Experimental results show that DELTA not only saves 40%-70% of GPU memory, surpassing the state-of-the-art method to a great extent but also gets comparable convergence results as the baseline with acceptable time delay. Also, DELTA gains 2.04$\times$ maximum batchsize when training ResNet-50 and 2.25$\times$ when training ResNet-101 compared with the baseline. Besides, comparisons between the swapping cost and recomputation cost in our experiments demonstrate the importance of making a reasonable dynamic scheduler on tensor swapping and tensor recomputation, which refutes the arguments in some related work that swapping should be the first and best choice.
Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Training
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Shanghai Jiao Tong University（上海交通大学）；Huawei Cloud（华为云）；Zhejiang Lab（浙江实验室）
SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks
Brown University（布朗大学）；Univ. of Electr. Sci. & Tech. of China（电子科技大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Pacific Northwest National Laboratory（太平洋西北国家实验室）；Massachusetts Institute of Technology（麻省理工学院）
GPU Memory Usage Optimization for Backward Propagation in Deep Network Training
Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）；Department of Computer Science and Information Engineering, National Taiwan University（台湾大学资讯工程学系）
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
City University of Hong Kong（香港城市大学）；Harvard University（哈佛大学）；Zhejiang University（浙江大学）；NLPR & MAIS, Institute of Automation, CAS（中国科学院自动化研究所模式识别国家重点实验室与多模态人工智能系统全国重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
