arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.18596cs.LGcs.AI
收藏

DeltaLLM:利用共享权重间的低秩差值压缩大语言模型

DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights

Microsoft(微软公司)

Liana Mikaelyan, Ayyoob Imani, Mathew Salvaris, Parth Pathak, Mohsen Fayyaz

AI总结:

提出DeltaLLM,通过层间权重共享加低秩差值矩阵的结构,以少量token轻量训练实现12%参数缩减并保留90%性能,优于现有压缩方法。

摘要:

我们提出了DeltaLLM,一种新的训练后压缩技术,旨在减少大语言模型(LLM)的内存占用。我们提出了一种替代性的LLM结构组织方式,在相邻Transformer块之间的层间实现权重共享,并在它们之间引入额外的低秩差值矩阵。在训练方面,我们采用了渐进式模块替换方法,并表明仅使用约3000万至4000万个token对低秩模块进行轻量级训练,就足以达到与从头训练的同规模LLM相当的性能。我们发布了由此产生的模型DeltaLLAMA和DeltaPHI,参数减少了12%,在常见知识和推理基准上保留了基础Llama和Phi模型90%的性能。我们的方法还优于在移除相同数量参数情况下的压缩技术JointDrop、LaCo、ShortGPT和SliceGPT。例如,参数减少24%的DeltaPhi 2.9B,其平均零样本准确率与经过恢复微调、参数减少12%的SlicedPhi 3.3B相当,尽管前者在未进行任何微调的情况下小了约4亿参数。这项工作为存储空间至关重要时的大语言模型架构设计和压缩方法提供了新的见解。

英文摘要:

We introduce DeltaLLM, a new post-training compression technique to reduce the memory footprint of LLMs. We propose an alternative way of structuring LLMs with weight sharing between layers in subsequent Transformer blocks, along with additional low-rank difference matrices between them. For training, we adopt the progressing module replacement method and show that the lightweight training of the low-rank modules with approximately 30M-40M tokens is sufficient to achieve performance on par with LLMs of comparable sizes trained from scratch. We release the resultant models, DeltaLLAMA and DeltaPHI, with a 12% parameter reduction, retaining 90% of the performance of the base Llama and Phi models on common knowledge and reasoning benchmarks. Our method also outperforms compression techniques JointDrop, LaCo, ShortGPT and SliceGPT with the same number of parameters removed. For example, DeltaPhi 2.9B with a 24% reduction achieves similar average zero-shot accuracies as recovery fine-tuned SlicedPhi 3.3B with a 12% reduction, despite being approximately 400M parameters smaller with no fine-tuning applied. This work provides new insights into LLM architecture design and compression methods when storage space is critical.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑