Small language models (SLMs) have attracted considerable attention from both academia and industry due to their broad range of applications in edge devices. To obtain SLMs with strong performance, conventional approaches either pre-train the models from scratch, which incurs substantial computational costs, or compress/prune existing large language models (LLMs), which results in performance drops and falls short in comparison to pre-training. In this paper, we investigate the family of acceleration methods that involve both structured pruning and model training. We found 1) layer-wise adaptive pruning (Adapt-Pruner) is extremely effective in LLMs and yields significant improvements over existing pruning techniques, 2) adaptive pruning equipped with further training leads to models comparable to those pre-training from scratch, 3) incremental pruning brings non-trivial performance gain by interleaving pruning with training and only removing a small portion of neurons ($\sim$5%) at a time. Experimental results on LLaMA-3.1-8B demonstrate that Adapt-Pruner outperforms conventional pruning methods, such as LLM-Pruner, FLAP, and SliceGPT, by an average of 1%-7% in accuracy on commonsense benchmarks. Additionally, Adapt-Pruner restores the performance of MobileLLM-125M to 600M on the MMLU benchmark with 200$\times$ fewer tokens via pruning from its larger counterparts, and discovers a new 1B model that surpasses LLaMA-3.2-1B in multiple benchmarks. The official code is released at https://github.com/research4pan/AdaptPruner.
Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retraining
Department of Electrical and Computer Engineering, North South University, Dhaka, 1229, Bangladesh（电气与计算机工程系，北南大学，达卡，孟加拉国）；Data Science and AI, University of Doha for Science and Technology, Doha, Qatar（数据科学与人工智能，多哈科学技术大学，多哈，卡塔尔）
Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Hong Kong Baptist Univeristy（香港浸会大学）；Harbin Institude of Technology, Shenzhen（哈尔滨工业大学（深圳））
E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models
Digital Environment Research Institute (DERI), Queen Mary University of London（伦敦大学玛丽女王学院数字环境研究所（DERI））；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海人工智能教育研究院）；National Institute of Education, Nanyang Technological University（南洋理工大学国家教育学院）；School of Computer Science and Engineering, Jiangsu University of Science and Technology（江苏科技大学计算机科学与工程学院）
O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning
China Agriculture University（中国农业大学）；Tsinghua University（清华大学）；University of Oxford（牛津大学）；Didichuxing Co. Ltd（滴滴出行有限公司）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Nanyang Technological University（南洋理工大学）
Persona-Pruner: Sculpting Lightweight Models for Role-Playing
Department of Artificial Intelligence, Korea University, Seoul, South Korea（韩国大学人工智能系）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi（德里印度理工学院亚迪人工智能学院）；Indian Institute of Technology Delhi（德里印度理工学院）；Microsoft Research (MSR) India（微软研究院印度分部）
Less is More: Towards Green Code Large Language Models via Unified Structural Pruning
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Singapore Management University（新加坡国立管理学院）；National University of Defense Technology（国防科技大学）；Nantong University（南通大学）；Monash University and CSIRO’s Data61（莫纳什大学和CSIRO Data61）；Birkbeck, University of London（伦敦大学伯贝克学院）
FAIR-Pruner: A Flexible Framework for Automatic Layer-Wise Pruning via Tolerance of Difference
School of Statistics and Mathematics, Zhejiang Gongshang University（浙江工商大学统计与数学学院）；École de technologie supérieure (ÉTS), Université du Québec（魁北克大学埃克森技术学院）；Southern University of Science and Technology（南方科技大学）
Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）
Fluctuation-based Adaptive Structured Pruning for Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Wuhan AI Research（武汉人工智能研究院）；Objecteye Inc.（Objecteye公司）
Technical University of Vienna（维也纳技术大学）；MIT（麻省理工学院）
In this paper, we present a novel sensitivity-based filter pruning algorithm (SbF-Pruner) to learn the importance scores of filters of each layer end-to-end. Our method learns the scores from the filter weights, enabling it to account for the correlations between the filters of each layer. Moreover, by training the pruning scores of all layers simultaneously our method can account for layer interdependencies, which is essential to find a performant sparse sub-network. Our proposed method can train and generate a pruned network from scratch in a straightforward, one-stage training process witho…
Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning…
We present a filter pruning approach for deep model compression, using a multitask network. Our approach is based on learning a a pruner network to prune a pre-trained target network. The pruner is essentially a multitask deep neural network with binary outputs that help identify the filters from each layer of the original network that do not have any significant contribution to the model and can therefore be pruned. The pruner network has the same architecture as the original network except that it has a multitask/multi-output last layer containing binary-valued outputs (one per filter), whic…
Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs
School of Electronics and Computer Science, University of Southampton, UK（索姆塞特大学电子与计算机科学学院）；Department of Computer Science, University of Liverpool, UK（利弗pool大学计算机科学系）
Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study this question through the lens of model compression. We present a generic, structured pruning approach by parameterizing each weight matrix using its low-rank factorization, and adaptively removing rank-1 components during training. On language modeling tasks, our structured app…
