arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.17660cs.LG
收藏

Grass:基于结构化稀疏梯度的计算高效低内存LLM训练

Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients

Carnegie Mellon University(卡内基梅隆大学)

Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, Virginia Smith

AI总结:

本文提出Grass方法,利用稀疏投影将梯度转化为结构化稀疏更新,显著降低LLM训练的内存、计算与通信开销,在单张40GB A100上实现13B LLaMA半精度预训练并提升2倍吞吐量。

摘要:

大型语言模型(LLM)的训练和微调通常受限于有限的GPU内存。现有的基于投影的优化方法通过将梯度投影到低维子空间来减少优化器状态内存,但它们通常依赖密集投影矩阵,这会引入计算和内存开销。在本研究中,我们提出了Grass(GRAdient Stuctured Sparsification),一种利用稀疏投影将梯度转化为结构化稀疏更新的新方法。该设计不仅显著减少了优化器状态的内存使用,还最小化了梯度内存占用、计算和通信成本,从而大幅提升吞吐量。在预训练和微调任务上的大量实验表明,Grass实现了与全秩训练及现有基于投影方法相媲美的性能。值得注意的是,Grass实现了在单张40GB A100 GPU上对130亿参数LLaMA模型进行半精度预训练——这是以往方法无法实现的壮举——并在8-GPU系统上实现了高达2倍的吞吐量提升。代码可在 https://github.com/aashiqmuhamed/GRASS 找到。

英文摘要:

Large language model (LLM) training and finetuning are often bottlenecked by limited GPU memory. While existing projection-based optimization methods address this by projecting gradients into a lower-dimensional subspace to reduce optimizer state memory, they typically rely on dense projection matrices, which can introduce computational and memory overheads. In this work, we propose Grass (GRAdient Stuctured Sparsification), a novel approach that leverages sparse projections to transform gradients into structured sparse updates. This design not only significantly reduces memory usage for optimizer states but also minimizes gradient memory footprint, computation, and communication costs, leading to substantial throughput improvements. Extensive experiments on pretraining and finetuning tasks demonstrate that Grass achieves competitive performance to full-rank training and existing projection-based methods. Notably, Grass enables half-precision pretraining of a 13B parameter LLaMA model on a single 40GB A100 GPU--a feat infeasible for previous methods--and yields up to a $2\times$ throughput improvement on an 8-GPU system. Code can be found at https://github.com/aashiqmuhamed/GRASS .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑