arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.12499cs.LGcs.DS
收藏

深度网络训练中反向传播的 GPU 显存使用优化

GPU Memory Usage Optimization for Backward Propagation in Deep Network Training

Institute of Information Science, Academia Sinica(中央研究院资讯科学研究所);Department of Computer Science and Information Engineering, National Taiwan University(台湾大学资讯工程学系)

Ding-Yong Hong, Tzu-Hsien Tsai, Ning Wang, Pangfeng Liu, Jan-Jan Wu

AI总结:

针对深度网络训练中卷积层中间数据导致的 GPU 显存压力,本文形式化检查点选择问题,先提出 O(n³) 动态规划算法,再结合实验追踪修正目标函数,给出 O(n) 最优重计算检查点选择算法。

摘要:

在现代深度学习中,为执行更复杂的任务并获得更好的准确率,设计更大的深度神经网络(DNN)已成为一种趋势。另一方面,卷积神经网络(CNN)已成为大多数计算机视觉任务的标准方法。然而,在模型训练期间,卷积层中间数据的内存分配可能会造成严重的显存压力。人们已经提出了许多解决方案来解决这一问题。除了依赖硬件的解决方案之外,一种通用方法——重计算(rematerialization)——能够通过以计算换取内存来高效降低 GPU 显存使用。其思想是在前向阶段选择一组中间结果作为检查点,并且只将它们保存在内存中,以减少显存占用。反向阶段则根据需要,从内存中最近的检查点重新计算中间数据。这种重新计算会增加执行时间,但由于在前向阶段不将所有中间结果都存储在内存中,因而节省了显存。本文中,我们将重点关注如何高效找到最优检查点子集,以在模型训练期间实现最低的峰值显存使用。我们首先用数学方程描述神经网络训练的理论背景。我们利用这些方程识别在前向和反向阶段计算模型权重梯度所需的全部必要数据。我们首先明确了检查点选择问题,并提出一种时间复杂度为 O(n³) 的动态规划算法,以解决寻找最优检查点子集的问题。在大量实验的基础上,我们利用理论分析对该问题给出了更准确的描述,并根据追踪结果修正了目标函数,进而提出一种用于寻找最优检查点子集的 O(n) 时间算法。

英文摘要:

In modern Deep Learning, it has been a trend to design larger Deep Neural Networks (DNNs) for the execution of more complex tasks and better accuracy. On the other hand, Convolutional Neural Networks (CNNs) have become the standard method for most of computer vision tasks. However, the memory allocation for the intermediate data in convolution layers can cause severe memory pressure during model training. Many solutions have been proposed to resolve the problem. Besides hardware-dependent solutions, a general methodology rematerialization can reduce GPU memory usage by trading computation for memory efficiently. The idea is to select a set of intermediate results during the forward phase as checkpoints, and only save them in memory to reduce memory usage. The backward phase recomputes the intermediate data from the closest checkpoints in memory as needed. This recomputation increases execution time but saves memory by not storing all intermediate results in memory during the forward phase. In this paper, we will focus on efficiently finding the optimal checkpoint subset to achieve the least peak memory usage during the model training. We first describe the theoretical background of the training of a neural network using mathematical equations. We use these equations to identify all essential data required during both forward and backward phases to compute the gradient of weights of the model. We first identify the checkpoint selection problem and propose a dynamic programming algorithm with time complexity O(n3) to solve the problem of finding the optimal checkpoint subset. With extensive experiments, we formulate a more accurate description of the problem using our theoretical analysis and revise the objective function based on the tracing, and propose an O(n)-time algorithm for finding the optimal checkpoint subset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑