Transformer models have revolutionized natural language processing with their unparalleled ability to grasp complex contextual relationships. However, the vast number of parameters in these models has raised concerns regarding computational efficiency, environmental impact, and deployability on resource-limited platforms. To address these challenges, this paper investigates the application of weight pruning-a strategic reduction of model parameters based on their significance-as an optimization strategy for Transformer architectures. Through extensive experimentation, we explore various pruning methodologies, highlighting their impact on model performance, size, and computational demands. Our findings suggest that with judicious selection of pruning hyperparameters, significant reductions in model size are attainable without considerable compromise on performance. Moreover, when coupled with post-pruning fine-tuning strategies, some pruned models even exhibit enhanced generalization capabilities. This work seeks to bridge the gap between model efficiency and performance, paving the way for more scalable and environmentally responsible deep learning applications.
On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning
Department of Computer Science, TU Dortmund University（图恩-多特蒙德大学计算机科学系）；Bonn-Aachen International Center for Information Technology, University of Bonn（波恩-亚琛国际信息科技中心，波恩大学）；Computer Science School, University of Sheffield（谢菲尔德大学计算机科学学院）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔人工智能与机器学习研究所）
Auto-regressive large language models such as GPT-3 require enormous computational resources to use. Traditionally, structured pruning methods are employed to reduce resource usage. However, their application to and efficacy for generative language models is heavily under-explored. In this paper we conduct an comprehensive evaluation of common structured pruning methods, including magnitude, random, and movement pruning on the feed-forward layers in GPT-type models. Unexpectedly, random pruning results in performance that is comparable to the best established methods, across multiple natural l…
University of Connecticut（康涅狄格大学）；Northeastern University（东北大学）；Syracuse University（雪城大学）；Microsoft Corporation（微软公司）；Stevens Institute of Technology（史蒂文斯理工学院）
Pre-trained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and computational speed on hardware platforms have impeded the popularity of pre-trained models, especially in the era of edge computing. In this work, we propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. We incorporate the reweighted group Lasso into block-structured pruning for optimization. Besides the significantly reduced weight storage and computat…
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
The University of Hong Kong（香港大学）；OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室开放智能研究院）；Simon Fraser University（西蒙菲莎大学）；Southern University of Science and Technology（南方科技大学）
Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study this question through the lens of model compression. We present a generic, structured pruning approach by parameterizing each weight matrix using its low-rank factorization, and adaptively removing rank-1 components during training. On language modeling tasks, our structured app…
PAT: Pruning-Aware Tuning for Large Language Models
School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；University of Arizona（亚利桑那大学）；Samsung Electronic Research Centre of China（三星中国电子研究中心）；Interdisciplinary Research Center for Future Intelligent Chips, Nanjing University（南京大学未来智能芯片交叉研究中心）
Efficient Post-Training Pruning of Large Language Models with Statistical Correction
Department of Computer Science and Engineering, Santa Clara University, Santa Clara, CA, USA（计算机科学与工程系，圣克拉拉大学）；Futurewei Technologies, Inc., USA（未来韦艾技术公司）
iFLYTEK Research（科大讯飞研究院）；Harbin Institute of Technology（哈尔滨工业大学）
Pre-trained language models have been prevailed in natural language processing and become the backbones of many NLP tasks, but the demands for computational resources have limited their applications. In this paper, we introduce TextPruner, an open-source model pruning toolkit designed for pre-trained language models, targeting fast and easy model compression. TextPruner offers structured post-training pruning methods, including vocabulary pruning and transformer pruning, and can be applied to various models and tasks. We also propose a self-supervised pruning method that can be applied without…
Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability
organization= College Of Science \& Engineering, Hamad Bin Khalifa University , addressline= Education City , city= Doha , postcode= 34110 , country= Qatar
Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning…
Fluctuation-based Adaptive Structured Pruning for Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Wuhan AI Research（武汉人工智能研究院）；Objecteye Inc.（Objecteye公司）
Pre-trained Language Models (PLMs) have achieved great success in various Natural Language Processing (NLP) tasks under the pre-training and fine-tuning paradigm. With large quantities of parameters, PLMs are computation-intensive and resource-hungry. Hence, model pruning has been introduced to compress large-scale PLMs. However, most prior approaches only consider task-specific knowledge towards downstream tasks, but ignore the essential task-agnostic knowledge during pruning, which may cause catastrophic forgetting problem and lead to poor generalization ability. To maintain both task-agnost…
Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Hong Kong Baptist Univeristy（香港浸会大学）；Harbin Institude of Technology, Shenzhen（哈尔滨工业大学（深圳））
Frustratingly Easy Task-aware Pruning for Large Language Models
Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；University of Science and Technology of China（中国科学技术大学）；Xi’an Jiaotong University（西安交通大学）
Institute of Science and Technology Austria（奥地利科学技术研究所）；Neural Magic Inc.（Neural Magic公司）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Transformer-based language models have become a key building block for natural language processing. While these models are extremely accurate, they can be too large and computationally intensive to run on standard deployments. A variety of compression methods, including distillation, quantization, structured and unstructured pruning are known to decrease model size and increase inference speed, with low accuracy loss. In this context, this paper's contributions are two-fold. We perform an in-depth study of the accuracy-compression trade-off for unstructured weight pruning of BERT models. We in…
Pruning General Large Language Models into Customized Expert Models
National University of Singapore（国立新加坡大学）；Nanyang Technological University（南洋理工大学）；DAMO Academy, Alibaba Group（阿里达摩院）；MiroMind；Singapore University of Technology and Design（新加坡科技设计大学）
Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning…
Exploring Federated Pruning for Large Language Models
The University of Hong Kong（香港大学）；Southern University of Science and Technology（南方科技大学）；Sun Yat-sen University（中山大学）；Guangming Laboratory（光明实验室）；Hangzhou Dianzi University（杭州电子科技大学）
Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models
Shenzhen Future Network of Intelligence Institute（深圳未来网络智能研究院）；Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来网络智能重点实验室）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Beijing Normal University（北京师范大学）；Alibaba Group（阿里巴巴集团）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）
IG-Pruning: Input-Guided Block Pruning for Large Language Models
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）；Key Laboratory of AI Safety, Chinese Academy of Sciences（人工智能安全重点实验室，中国科学院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
