arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.07378cs.CLcs.LG
收藏

SVD-LLM:面向大语言模型压缩的截断感知奇异值分解

SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

The Ohio State University(俄亥俄州立大学);Michigan State University(密歇根州立大学)

Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang

AI总结:

本文提出训练后压缩方法SVD-LLM,通过截断感知数据白化和序贯低秩参数更新降低SVD压缩损失,并在多家族、多规模模型与10个数据集上验证其高压缩比优势。

摘要:

大语言模型(LLM)的进展受到其庞大体积的阻碍,因此实际部署需要LLM压缩方法。奇异值分解(SVD)为LLM压缩提供了一种有前景的解决方案。然而,基于SVD的最先进LLM压缩方法存在两个关键局限:截断较小的奇异值可能导致更高的压缩损失,并且在SVD截断后缺乏对压缩权重的更新。在本工作中,我们提出SVD-LLM,这是一种基于SVD的训练后LLM压缩方法,解决了现有方法的局限。SVD-LLM引入截断感知数据白化技术,以确保奇异值与压缩损失之间的直接映射。此外,SVD-LLM采用带序贯低秩近似的参数更新,以补偿SVD压缩后的准确率下降。我们在10个数据集以及来自三个不同LLM家族、三种不同规模的七个模型上评估了SVD-LLM。结果证明SVD-LLM优于最先进方法,尤其是在高模型压缩比下。代码可在https://github.com/AIoT-MLSys-Lab/SVD-LLM获取。

英文摘要:

The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitates LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression. However, state-of-the-art SVD-based LLM compression methods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weights after SVD truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-aware data whitening technique to ensure a direct mapping between singular values and compression loss. Moreover, SVD-LLM adopts a parameter update with sequential low-rank approximation to compensate for the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10 datasets and seven models from three different LLM families at three different scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts, especially at high model compression ratios. Our code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑