Light-PEFT:通过早期剪枝轻量化参数高效微调
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning
AI总结:
Light-PEFT 框架通过在训练早期对基础模型和 PEFT 模块进行掩码与多粒度剪枝,在保持性能的同时将基础模型参数剪枝超 40%、可训练参数降至原 PEFT 的 25%,实现训练与推理加速并减少内存占用。
摘要:
参数高效微调(PEFT)已成为大语言模型时代微调的主流技术。然而,现有 PEFT 方法的训练效率仍然不足。首先,在训练过程中使用大规模基础模型对某些微调任务而言存在过度冗余。其次,随着模型规模增大,经验性添加的 PEFT 模块中可训练参数的增长变得不可忽略且冗余,导致效率低下。为了实现任务特定的高效微调,我们提出了 Light-PEFT 框架,该框架包含两种方法:基础模型的掩码早期剪枝和 PEFT 的多粒度早期剪枝。Light-PEFT 框架允许在训练早期同时估计基础模型和 PEFT 模块中的冗余参数,然后可以剪枝这些参数以实现更高效的微调。我们在 GLUE、SuperGLUE、问答任务以及多种模型上验证了我们的方法。借助 Light-PEFT,基础模型的参数最多可剪枝超过 40%,同时仍将可训练参数控制为原始 PEFT 方法的 25%。与直接使用 PEFT 方法相比,Light-PEFT 实现了训练和推理加速,减少了内存占用,并保持了可比的性能以及 PEFT 的即插即用特性。
英文摘要:
Parameter-efficient fine-tuning (PEFT) has emerged as the predominant technique for fine-tuning in the era of large language models. However, existing PEFT methods still have inadequate training efficiency. Firstly, the utilization of large-scale foundation models during the training process is excessively redundant for certain fine-tuning tasks. Secondly, as the model size increases, the growth in trainable parameters of empirically added PEFT modules becomes non-negligible and redundant, leading to inefficiency. To achieve task-specific efficient fine-tuning, we propose the Light-PEFT framework, which includes two methods: Masked Early Pruning of the Foundation Model and Multi-Granularity Early Pruning of PEFT. The Light-PEFT framework allows for the simultaneous estimation of redundant parameters in both the foundation model and PEFT modules during the early stage of training. These parameters can then be pruned for more efficient fine-tuning. We validate our approach on GLUE, SuperGLUE, QA tasks, and various models. With Light-PEFT, parameters of the foundation model can be pruned by up to over 40%, while still controlling trainable parameters to be only 25% of the original PEFT method. Compared to utilizing the PEFT method directly, Light-PEFT achieves training and inference speedup, reduces memory usage, and maintains comparable performance and the plug-and-play feature of PEFT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。