arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.15092cs.CL
收藏

面向大语言模型压缩的奇异谱优化

Optimizing Singular Spectrum for Large Language Model Compression

Dengjie Li, Tiancheng Shen, Yao Zhou, Bernard Ghanem, Yibo Yang, Ming-Hsuan Yang

AI总结:

针对现有基于SVD的大语言模型压缩方法按奇异值排序分量与下游任务性能不匹配的问题,提出SoCo框架,通过可学习对角矩阵和三阶段训练自适应优化奇异谱,实现了更优的压缩-性能平衡。

摘要:

大语言模型(LLM)已展现出卓越的能力,但其极高的参数复杂度往往阻碍了部署。现有基于奇异值分解(SVD)的压缩方法仅将奇异值视为分解分量的重要性得分。然而,这种按奇异值排序的重要性不一定与下游任务的性能相关。在本研究中,我们提出了SoCo(面向大语言模型压缩的奇异谱优化)——一种新型压缩框架,它以数据驱动的方式学习重新缩放SVD的分解分量。具体而言,我们采用可学习对角矩阵为奇异谱分配重要性得分,并设计了三阶段训练流程,从初始粗压缩到细粒度稀疏化逐步优化这些得分,从而在大幅模型压缩与性能保留之间实现有效平衡。得益于可学习奇异谱,SoCo根据稀疏化后的重要性得分自适应剪枝分量,而非依赖奇异值的固定排序。更重要的是,剩余的重要性得分被放大的分量可以补偿被剪枝分量带来的损失。在多个大语言模型和基准上的实验评估表明,SoCo在模型压缩方面超越了当前最优方法。

英文摘要:

Large language models (LLMs) have demonstrated remarkable capabilities, yet prohibitive parameter complexity often hinders their deployment. Existing singular value decomposition (SVD) based compression methods simply deem singular values as importance scores of decomposed components. However, this importance ordered by singular values does not necessarily correlate with the performance of a downstream task. In this work, we introduce SoCo (Singular spectrum optimization for large language model Compression), a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner. Concretely, we employ a learnable diagonal matrix to assign importance scores for singular spectrum and develop a three-stage training process that progressively refines these scores from initial coarse compression to fine-grained sparsification-thereby striking an effective balance between aggressive model compression and performance preservation. Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores, rather than relying on the fixed order of singular values. More importantly, the remaining components with amplified importance scores can compensate for the loss of the pruned ones. Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑