arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11257cs.LG
收藏

ExCP:通过权重-动量联合收缩实现极致 LLM 检查点压缩

ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking

Huawei Noah’s Ark Lab(华为诺亚方舟实验室);University of Science and Technology of China(中国科学技术大学)

Wenshuo Li, Xinghao Chen, Han Shu, Yehui Tang, Yunhe Wang

AI总结:

针对 LLM 训练检查点存储开销问题,提出 ExCP 框架,通过相邻残差、权重-动量联合收缩和非均匀量化实现近无损极致压缩。

摘要:

大型语言模型(LLM)最近在人工智能领域引起了广泛关注。然而,这些模型的训练过程在计算和存储容量方面带来了巨大挑战,因此压缩检查点已成为一个紧迫的问题。本文提出了一种新颖的极致检查点压缩(ExCP)框架,可显著减少训练检查点所需的存储空间,同时实现近乎无损的性能。我们首先计算相邻检查点的残差,以获得关键但稀疏的信息,从而实现更高的压缩率。为进一步挖掘检查点中的冗余参数,我们随后提出权重-动量联合收缩方法,以利用模型优化过程中的另一项重要信息,即动量。具体而言,我们同时利用模型和优化器的信息,在保留关键信息以确保最佳性能的同时,尽可能多地丢弃参数。此外,我们利用非均匀量化进一步压缩检查点存储。我们在多个参数量从 410M 到 7B 的模型上对所提出的 ExCP 框架进行了广泛评估,结果表明其在保持强劲性能的同时显著减少了存储。例如,我们在 Pythia-410M 模型上实现了约 70 倍压缩,并且其在各种下游任务上的最终性能与原始模型一样准确。代码将在 https://github.com/Gaffey/ExCP 提供。

英文摘要:

Large language models (LLM) have recently attracted significant attention in the field of artificial intelligence. However, the training process of these models poses significant challenges in terms of computational and storage capacities, thus compressing checkpoints has become an urgent problem. In this paper, we propose a novel Extreme Checkpoint Compression (ExCP) framework, which significantly reduces the required storage of training checkpoints while achieving nearly lossless performance. We first calculate the residuals of adjacent checkpoints to obtain the essential but sparse information for higher compression ratio. To further excavate the redundancy parameters in checkpoints, we then propose a weight-momentum joint shrinking method to utilize another important information during the model optimization, i.e., momentum. In particular, we exploit the information of both model and optimizer to discard as many parameters as possible while preserving critical information to ensure optimal performance. Furthermore, we utilize non-uniform quantization to further compress the storage of checkpoints. We extensively evaluate our proposed ExCP framework on several models ranging from 410M to 7B parameters and demonstrate significant storage reduction while maintaining strong performance. For instance, we achieve approximately $70\times$ compression for the Pythia-410M model, with the final performance being as accurate as the original model on various downstream tasks. Codes will be available at https://github.com/Gaffey/ExCP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑