arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.09009cs.LGcs.CL
收藏

在大词汇量语言模型中减少损失

Cut Your Losses in Large-Vocabulary Language Models

Apple(苹果公司)

Erik Wijmans, Brody Huval, Alexander Hertzberg, Vladlen Koltun, Philipp Krähenbühl

AI总结:

本文提出Cut Cross-Entropy(CCE)方法,通过仅计算正确token的logit并利用闪存进行log-sum-exp归约,大幅降低大词汇量语言模型训练中交叉熵损失的内存占用,同时保持训练速度和收敛性。

摘要:

随着语言模型规模不断扩大,其词汇表也在不断扩大。这使得LLM在训练期间的内存占用不成比例地转移到了单一层:损失计算中的交叉熵。交叉熵会构建一个logit矩阵,其中包含每对输入token与词汇表项的条目,对于小型模型而言,其消耗的内存比LLM其余部分的总和多出一个数量级。我们提出Cut Cross-Entropy(CCE),一种在不将所有token的logit具体化到全局内存的情况下计算交叉熵损失的方法。相反,CCE仅计算正确token的logit,并即时评估所有logit上的log-sum-exp。我们实现了一个自定义内核,在闪存中执行矩阵乘法和词汇表上的log-sum-exp归约,使得交叉熵计算的全局内存消耗可以忽略不计。这产生了显著的效果。以Gemma 2(2B)模型为例,CCE将损失计算的内存占用从24 GB降低到1 MB,并将分类器头在训练期间的总内存消耗从28 GB降低到1 GB。为了提高CCE的吞吐量,我们利用softmax固有的稀疏性,并提出跳过梯度计算中对梯度贡献可忽略不计(即低于数值精度)的元素。实验表明,内存消耗的大幅降低是在不牺牲训练速度或收敛性的情况下实现的。

英文摘要:

As language models grow ever larger, so do their vocabularies. This has shifted the memory footprint of LLMs during training disproportionately to one single layer: the cross-entropy in the loss computation. Cross-entropy builds up a logit matrix with entries for each pair of input tokens and vocabulary items and, for small models, consumes an order of magnitude more memory than the rest of the LLM combined. We propose Cut Cross-Entropy (CCE), a method that computes the cross-entropy loss without materializing the logits for all tokens into global memory. Rather, CCE only computes the logit for the correct token and evaluates the log-sum-exp over all logits on the fly. We implement a custom kernel that performs the matrix multiplications and the log-sum-exp reduction over the vocabulary in flash memory, making global memory consumption for the cross-entropy computation negligible. This has a dramatic effect. Taking the Gemma 2 (2B) model as an example, CCE reduces the memory footprint of the loss computation from 24 GB to 1 MB, and the total training-time memory consumption of the classifier head from 28 GB to 1 GB. To improve the throughput of CCE, we leverage the inherent sparsity of softmax and propose to skip elements of the gradient computation that have a negligible (i.e., below numerical precision) contribution to the gradient. Experiments demonstrate that the dramatic reduction in memory consumption is accomplished without sacrificing training speed or convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑