DropBP：通过丢弃反向传播加速大型语言模型的微调
DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation
AI总结：
DropBP通过在反向传播中随机丢弃层并基于层敏感性分配丢弃率，降低计算和内存成本，同时保持精度，可与PEFT结合，显著加速LLM微调。
摘要：
大型语言模型（LLMs）已在各个领域取得了显著成功。然而，训练这些LLMs在前向和反向传播过程中通常涉及大量的内存和计算成本。虽然参数高效微调（PEFT）显著减少了与参数相关的训练内存，但它并未解决显著的计算成本和激活内存问题。在本文中，我们提出了丢弃反向传播（DropBP），一种旨在降低计算成本和激活内存同时保持准确性的新方法。DropBP在反向传播过程中随机丢弃层，这本质上等同于训练由未丢弃层和残差连接生成的浅层子模块。此外，DropBP计算每层的敏感性以分配适当的丢弃率，从而稳定训练过程。DropBP不仅适用于全量微调，还可以通过在后向传播过程中丢弃层，与所有类型的PEFT正交集成。具体来说，DropBP可以将训练时间减少44%，同时保持与基线相当的准确性，将收敛到相同困惑度的速度加速1.5倍，并支持在单个NVIDIA-A100 GPU上以6.2倍更大的序列长度进行训练。此外，我们的DropBP在NVIDIA A100 GPU上实现了79%的吞吐量提升，在Intel Gaudi2 HPU上实现了117%的提升。代码可在https://github.com/WooSunghyeon/dropbp获取。
英文摘要：
Large language models (LLMs) have achieved significant success across various domains. However, training these LLMs typically involves substantial memory and computational costs during both forward and backward propagation. While parameter-efficient fine-tuning (PEFT) considerably reduces the training memory associated with parameters, it does not address the significant computational costs and activation memory. In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs and activation memory while maintaining accuracy. DropBP randomly drops layers during backward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. Additionally, DropBP calculates the sensitivity of each layer to assign an appropriate drop rate, thereby stabilizing the training process. DropBP is not only applicable to full fine-tuning but can also be orthogonally integrated with all types of PEFT by dropping layers during backward propagation. Specifically, DropBP can reduce training time by 44% with comparable accuracy to the baseline, accelerate convergence to the same perplexity by 1.5x, and enable training with a sequence length 6.2x larger on a single NVIDIA-A100 GPU. Furthermore, our DropBP enabled a throughput increase of 79% on a NVIDIA A100 GPU and 117% on an Intel Gaudi2 HPU. The code is available at https://github.com/WooSunghyeon/dropbp.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
