Efficient Post-Training Pruning of Large Language Models with Statistical Correction
Department of Computer Science and Engineering, Santa Clara University, Santa Clara, CA, USA（计算机科学与工程系，圣克拉拉大学）；Futurewei Technologies, Inc., USA（未来韦艾技术公司）
Post-training pruning is an effective approach for reducing the size and inference cost of large language models (LLMs), but existing methods often face a trade-off between pruning quality and computational efficiency. Heuristic pruning methods are efficient but sensitive to activation outliers, while reconstruction-based approaches improve fidelity at the cost of heavy computation. In this work, we propose a lightweight post-training pruning framework based on first-order statistical properties of model weights and activations. During pruning, channel-wise statistics are used to calibrate magnitude-based importance scores, reducing bias from activation-dominated channels. After pruning, we apply an analytic energy compensation to correct distributional distortions caused by weight removal. Both steps operate without retraining, gradients, or second-order information. Experiments across multiple LLM families, sparsity patterns, and evaluation tasks show that the proposed approach improves pruning performance while maintaining computational cost comparable to heuristic methods. The results suggest that simple statistical corrections can be effective for post-training pruning of LLMs.
P$^2$ Law: Scaling Law for Post-Training After Model Pruning
Engineering Research Center of Database and Business Intelligence, MOE, China（数据库与商业智能工程研究中心，教育部，中国）；School of Information, Renmin University of China,Beijing, China（信息学院，中国人民大学，北京，中国）；Key Laboratory of Data Engineering and Knowledge Engineering, MOE, China（数据工程与知识工程重点实验室，教育部，中国）；Zhipu AI, China（智谱AI，中国）
Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retraining
Department of Electrical and Computer Engineering, North South University, Dhaka, 1229, Bangladesh（电气与计算机工程系，北南大学，达卡，孟加拉国）；Data Science and AI, University of Doha for Science and Technology, Doha, Qatar（数据科学与人工智能，多哈科学技术大学，多哈，卡塔尔）
Attribution-Guided Pruning for Insight and Control: Circuit Discovery and Targeted Correction in Small-scale LLMs
Data Science Center ScaDS.AI, Universität Leipzig（ScaDS.AI数据科学中心，莱比锡大学）；Department of Artificial Intelligence, Fraunhofer Heinrich-Hertz-Institute（人工智能系，弗劳恩霍夫 Heinrich-Hertz 研究所）；Department of Electrical Engineering and Computer Science, Technische Universität Berlin（电气工程与计算机科学系，柏林技术大学）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Centre of eXplainable Artificial Intelligence, Technological University Dublin（可解释人工智能中心，都柏林技术大学）
Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Hong Kong Baptist Univeristy（香港浸会大学）；Harbin Institude of Technology, Shenzhen（哈尔滨工业大学（深圳））
Beware of Calibration Data for Pruning Large Language Models
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Key Laboratory of Data Intelligence and Advanced Computing, Soochow University（苏州大学数据智能与先进计算重点实验室）；Huawei Cloud, China（华为云（中国））
iFLYTEK Research（科大讯飞研究院）；Harbin Institute of Technology（哈尔滨工业大学）
Pre-trained language models have been prevailed in natural language processing and become the backbones of many NLP tasks, but the demands for computational resources have limited their applications. In this paper, we introduce TextPruner, an open-source model pruning toolkit designed for pre-trained language models, targeting fast and easy model compression. TextPruner offers structured post-training pruning methods, including vocabulary pruning and transformer pruning, and can be applied to various models and tasks. We also propose a self-supervised pruning method that can be applied without…
On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning
Department of Computer Science, TU Dortmund University（图恩-多特蒙德大学计算机科学系）；Bonn-Aachen International Center for Information Technology, University of Bonn（波恩-亚琛国际信息科技中心，波恩大学）；Computer Science School, University of Sheffield（谢菲尔德大学计算机科学学院）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔人工智能与机器学习研究所）
Pruning General Large Language Models into Customized Expert Models
National University of Singapore（国立新加坡大学）；Nanyang Technological University（南洋理工大学）；DAMO Academy, Alibaba Group（阿里达摩院）；MiroMind；Singapore University of Technology and Design（新加坡科技设计大学）
LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models
School of Microelectronics, Southern University of Science and Technology（微电子学院，南方科技大学）；Department of Electrical and Electronic Engineering, University of Hong Kong（电气与电子工程系，香港大学）
Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Data Science, Fudan University（复旦大学数据科学学院）
PAT: Pruning-Aware Tuning for Large Language Models
School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；University of Arizona（亚利桑那大学）；Samsung Electronic Research Centre of China（三星中国电子研究中心）；Interdisciplinary Research Center for Future Intelligent Chips, Nanjing University（南京大学未来智能芯片交叉研究中心）
Debias-SparseGPT: Bias-Aware Pruning for Large Language Models
Laboratoire Hubert Curien（于贝尔·屈里安实验室）；Université Claude Bernard Lyon 1（里昂第一大学（克洛德·贝尔纳））；Université Lumière Lyon 2（里昂第二大学（吕米埃尔））；École Centrale de Lyon（里昂中央理工学院）
Pruning is an effective way to reduce the huge inference cost of Transformer models. However, prior work on pruning Transformers requires retraining the models. This can add high training cost and high complexity to model deployment, making it difficult to use in many practical situations. To address this, we propose a fast post-training pruning framework for Transformers that does not require any retraining. Given a resource constraint and a sample dataset, our framework automatically prunes the Transformer model using structured sparsity methods. To retain high accuracy without retraining, w…
Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning
Institute of Information Science, Academia Sinica（中央研究院資訊科學研究所）；National Taiwan University of Science and Technology（國立臺灣科技大學）；AI Research Center, Hon Hai Research Institute（鴻海研究院人工智能研究中心）
