arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16282cs.LGcs.AI
收藏

通过近似反向传播与内存共享反向传播降低微调内存开销

Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation

School of Statistics and Data Science, Nankai University(南开大学统计与数据科学学院);School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Department of Automation, Tsinghua University(清华大学自动化系);Central Research Institute, United Imaging Healthcare, Co., Ltd.(联影医疗股份有限公司中央研究院)

Yuchen Yang, Yingdong Shi, Cheems Wang, Xiantong Zhen, Yuxuan Shi, Jun Xu

AI总结:

针对大模型微调内存开销大的问题,提出近似反向传播理论与内存共享反向传播策略,在不增加计算、不降低效率的前提下,最多可降低约30%的峰值内存使用量。

摘要:

将预训练大模型微调至下游任务是一项重要问题,但由于参数规模庞大,该过程存在内存开销巨大的问题。本研究从激活函数和层归一化的角度出发,致力于降低微调过程中的内存开销。为此,我们提出了近似反向传播(Approx-BP)理论,为前向传播与反向传播的解耦提供了理论可行性。我们将Approx-BP理论应用于反向传播训练,推导出了GELU和SiLU激活函数的内存高效替代方案:在反向传播中使用ReLU的导数函数,同时保持前向传播不变。此外,我们提出了内存共享反向传播策略,使激活内存可在相邻两层之间共享,从而消除激活内存使用的冗余。我们的方法既不会引入额外计算,也不会降低训练效率。我们在预训练视觉模型和语言模型上开展了大量实验,结果表明,所提方法最多可降低约30%的峰值内存使用量。我们的代码已发布于https://github.com/yyyyychen/LowMemoryBP。

英文摘要:

Fine-tuning pretrained large models to downstream tasks is an important problem, which however suffers from huge memory overhead due to large-scale parameters. This work strives to reduce memory overhead in fine-tuning from perspectives of activation function and layer normalization. To this end, we propose the Approximate Backpropagation (Approx-BP) theory, which provides the theoretical feasibility of decoupling the forward and backward passes. We apply our Approx-BP theory to backpropagation training and derive memory-efficient alternatives of GELU and SiLU activation functions, which use derivative functions of ReLUs in the backward pass while keeping their forward pass unchanged. In addition, we introduce a Memory-Sharing Backpropagation strategy, which enables the activation memory to be shared by two adjacent layers, thereby removing activation memory usage redundancy. Our method neither induces extra computation nor reduces training efficiency. We conduct extensive experiments with pretrained vision and language models, and the results demonstrate that our proposal can reduce up to $\sim$$30\%$ of the peak memory usage. Our code is released at https://github.com/yyyyychen/LowMemoryBP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑