Large language models (LLMs) have demonstrated proficiency across various natural language processing (NLP) tasks but often require additional training, such as continual pre-training and supervised fine-tuning. However, the costs associated with this, primarily due to their large parameter count, remain high. This paper proposes leveraging \emph{sparsity} in pre-trained LLMs to expedite this training process. By observing sparsity in activated neurons during forward iterations, we identify the potential for computational speed-ups by excluding inactive neurons. We address associated challenges by extending existing neuron importance evaluation metrics and introducing a ladder omission rate scheduler. Our experiments on Llama-2 demonstrate that Sparsity-Accelerated Training (SAT) achieves comparable or superior performance to standard training while significantly accelerating the process. Specifically, SAT achieves a $45\%$ throughput improvement in continual pre-training and saves $38\%$ training time in supervised fine-tuning in practice. It offers a simple, hardware-agnostic, and easily deployable framework for additional LLM training. Our code is available at https://github.com/OpenDFM/SAT.
Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models
Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University(计算机科学与技术系、人工智能研究院、BNRist中心、THBI实验室、清华大学-博世联合机器学习中心、清华大学)
Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR via Supernet
Meta AI
从可穿戴设备到功能强大的智能设备,现代自动语音识别(ASR)模型可在各种计算预算不同的边缘设备上运行。为了在模型精度与模型大小的帕累托前沿进行导航,研究人员陷入了一个困境,即通过为每个边缘设备训练和微调模型来优化模型精度,同时保持训练GPU时间易于处理。在本文中,我们提出了全稀疏DNN,其中单个神经网络可以被修剪以生成针对大范围模型大小的优化模型。我们为全稀疏DNN开发了训练策略,使其能够沿着单词错误率(WER)与模型大小的帕累托前沿找到模型,同时保持训练GPU时间不超过训练一个奇异模型的时间。我们使用流式E2 E ASR模型演示了全稀疏DNN。我们的研究结果显示,与单独修剪的稀疏模型相比,LibriSpeech的训练时间和资源大大节省,准确率相似或更高:测试其他模型的WER提高了2%-6.6%。
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
Multimedia Laboratory (MMLab), The Chinese University of Hong Kong(香港中文大学多媒体实验室);Salesforce AI Research(Salesforce 人工智能研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);CPII under InnoHK(InnoHK 创新香港研发平台旗下产品可靠性及系统安全研发中心)