The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitates LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression. However, state-of-the-art SVD-based LLM compression methods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weights after SVD truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-aware data whitening technique to ensure a direct mapping between singular values and compression loss. Moreover, SVD-LLM adopts a parameter update with sequential low-rank approximation to compensate for the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10 datasets and seven models from three different LLM families at three different scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts, especially at high model compression ratios. Our code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
Department of Computer Science, Aalborg University Copenhagen(奥尔堡大学哥本哈根分校计算机科学系);MaLGa-DIBRIS, University of Genoa(热那亚大学MaLGa-DIBRIS);INFN, Sezione di Genova(国家核物理研究所热那亚分部);European Organization for Nuclear Research (CERN)(欧洲核子研究中心);Ceva, Inc.(Ceva公司)
SEER: Self-Enhancing Chain-of-Thought Compression for Reasoning Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(区块链与数据安全国家重点实验室,浙江大学);State Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学)
ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs
Shanghai University of Finance and Economics(上海财经大学);Southern University of Science and Technology(南方科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));The Chinese University of Hong Kong(香港中文大学)