arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01392cs.CL
收藏

大型语言模型的稀疏加速训练

Sparsity-Accelerated Training for Large Language Models

Shanghai Jiao Tong University(上海交通大学);Suzhou Laboratory(苏州实验室);AISpeech Co., Ltd.(思必驰科技股份有限公司)

Da Ma, Lu Chen, Pengyu Wang, Hongshen Xu, Hanqi Li, Liangtai Sun, Su Zhu, Shuai Fan, Kai Yu

AI总结:

本文提出稀疏加速训练(SAT)方法,通过利用预训练LLMs中激活神经元的稀疏性,结合扩展的神经元重要性评估指标和阶梯省略率调度器,在Llama-2上实现了持续预训练45%吞吐量提升和监督微调38%训练时间节省,且性能与标准训练相当或更优。

摘要:

大型语言模型(LLMs)已在多种自然语言处理(NLP)任务中展现出卓越能力,但通常需要额外训练,例如持续预训练和监督微调。然而,由于模型参数量巨大,相关成本仍然很高。本文提出利用预训练LLMs中的稀疏性来加速这一训练过程。通过观察前向迭代过程中被激活神经元的稀疏性,我们发现可以通过排除未激活神经元来实现计算加速。我们通过扩展现有的神经元重要性评估指标并引入阶梯省略率调度器来解决相关挑战。我们在Llama-2上的实验表明,稀疏加速训练(SAT)在显著加速训练过程的同时,取得了与标准训练相当或更优的性能。具体而言,SAT在持续预训练中实现了45%的吞吐量提升,并在监督微调中实际节省了38%的训练时间。它为额外的LLM训练提供了一个简单、与硬件无关且易于部署的框架。我们的代码可在https://github.com/OpenDFM/SAT获取。

英文摘要:

Large language models (LLMs) have demonstrated proficiency across various natural language processing (NLP) tasks but often require additional training, such as continual pre-training and supervised fine-tuning. However, the costs associated with this, primarily due to their large parameter count, remain high. This paper proposes leveraging \emph{sparsity} in pre-trained LLMs to expedite this training process. By observing sparsity in activated neurons during forward iterations, we identify the potential for computational speed-ups by excluding inactive neurons. We address associated challenges by extending existing neuron importance evaluation metrics and introducing a ladder omission rate scheduler. Our experiments on Llama-2 demonstrate that Sparsity-Accelerated Training (SAT) achieves comparable or superior performance to standard training while significantly accelerating the process. Specifically, SAT achieves a $45\%$ throughput improvement in continual pre-training and saves $38\%$ training time in supervised fine-tuning in practice. It offers a simple, hardware-agnostic, and easily deployable framework for additional LLM training. Our code is available at https://github.com/OpenDFM/SAT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑