In this paper, we address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective. Specifically, we identify a critical issue of ''$\textbf{reconstruction error explosion}$'' in existing LLMs sparsification methods. This refers to the cumulative effect of reconstruction errors throughout the sparsification process, where errors from earlier layers propagate and amplify in subsequent layers. As a result, the overall reconstruction error increases significantly, leading to a substantial degradation in model performance. Through theoretical analysis, we derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue. Our method uses a monotonically increasing arithmetic progression, reducing the process of determining sparsity rates for multiple layers to the determination of a single common difference hyperparameter. Remarkably, this allows for the optimal layer-wise sparsity rates to be identified with just a few trials. Both our theoretical analysis and experimental results demonstrate that this sparsity allocation scheme is near optimal. Extensive experiments show that our method significantly improves the performance of sparse LLMs across various architectures, outperforming existing layer-wise sparsity methods. Furthermore, it enhances the performance of various compression techniques and is applicable to vision and multimodal models. Notably, our method achieves a reduction of 52.10 in perplexity for the 70$\%$ sparse LLaMA2-7B model obtained via Wanda, improves average zero-shot accuracy by 10.50$\%$, and delivers speedups of 2.63$\times$ and 2.23$\times$ on CPU and GPU, respectively.
Adaptive Signal Resuscitation: Channel-wise Post-Pruning Repair for Sparse Vision Networks
Department of Mathematical and Statistical Sciences, Marquette University(马歇尔大学数学与统计科学系);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Cornell Ann S. Bowers College of Computing and Information Science, Cornell University(康奈尔大学安·S·博尔斯计算与信息科学学院)
Datum-Wise Classification: A Sequential Approach to Sparsity
Université Pierre et Marie Curie - UPMC(皮埃尔和玛丽·居里大学(巴黎第六大学));LIP6(巴黎第六大学计算机科学实验室);LIFL (UMR CNRS)(法国国家科学研究中心里尔基础信息学实验室);INRIA Lille Nord-Europe(法国国家信息与自动化研究所里尔-北欧分部);Université de Lille(里尔大学)
Layer-wise Model Pruning based on Mutual Information
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences(中国科学院智能信息处理重点实验室);Zhejiang University(浙江大学);Computer Center of Peking University(北京大学计算中心);Peng Cheng Laboratory(鹏城实验室)
Mix-and-Match Pruning: Globally Guided Layer-Wise Sparsification of DNNs
1 Brandenburg Technical University, Cottbus, Germany;2 University of Zanjan, Iran;3 Tallinn University of Technology, Estonia;4 Leibniz Institute for High Performance Microelectronics, Frankfurt Oder, Germany
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
The University of Hong Kong(香港大学);OpenGVLab, Shanghai AI Laboratory(上海人工智能实验室开放智能研究院);Simon Fraser University(西蒙菲莎大学);Southern University of Science and Technology(南方科技大学)
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
Multimedia Laboratory (MMLab), The Chinese University of Hong Kong(香港中文大学多媒体实验室);Salesforce AI Research(Salesforce 人工智能研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);CPII under InnoHK(InnoHK 创新香港研发平台旗下产品可靠性及系统安全研发中心)
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Nankai University(南开大学);Dartmouth College(达特茅斯学院);University of Oxford(牛津大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);International Computer Science Institute(国际计算机科学研究所);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);University of California at Berkeley(加州大学伯克利分校)