Large language models (LLMs) have demonstrated remarkable capabilities, yet prohibitive parameter complexity often hinders their deployment. Existing singular value decomposition (SVD) based compression methods simply deem singular values as importance scores of decomposed components. However, this importance ordered by singular values does not necessarily correlate with the performance of a downstream task. In this work, we introduce SoCo (Singular spectrum optimization for large language model Compression), a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner. Concretely, we employ a learnable diagonal matrix to assign importance scores for singular spectrum and develop a three-stage training process that progressively refines these scores from initial coarse compression to fine-grained sparsification-thereby striking an effective balance between aggressive model compression and performance preservation. Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores, rather than relying on the fixed order of singular values. More importantly, the remaining components with amplified importance scores can compensate for the loss of the pruned ones. Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.
NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainium
University of California, Merced USA;University of Wisconsin, Madison USA;Yotta Labs \& University of California, Merced USA;University of California, Merced;University of Wisconsin, Madison;Yotta Labs \& University of California, Merced
Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models
Tsinghua University(清华大学);Shanghai University of Finance and Economics(上海财经大学);Institute for AI, Tsinghua University(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Peking University(北京大学)
xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
Cornell University(康奈尔大学);University of Washington(华盛顿大学);Department of Computer Science, National Yang Ming Chiao Tung University(国立阳明交通大学计算机科学系);The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Surrey(塞夫顿大学);Microsoft Research Asia(微软亚洲研究院)
ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs
Shanghai University of Finance and Economics(上海财经大学);Southern University of Science and Technology(南方科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));The Chinese University of Hong Kong(香港中文大学)