Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning shows significant improvements in high-sparsity regimes. When combined with distillation, the approach achieves minimal accuracy loss with down to only 3% of the model parameters.
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
The University of Hong Kong（香港大学）；OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室开放智能研究院）；Simon Fraser University（西蒙菲莎大学）；Southern University of Science and Technology（南方科技大学）
Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Auto-regressive large language models such as GPT-3 require enormous computational resources to use. Traditionally, structured pruning methods are employed to reduce resource usage. However, their application to and efficacy for generative language models is heavily under-explored. In this paper we conduct an comprehensive evaluation of common structured pruning methods, including magnitude, random, and movement pruning on the feed-forward layers in GPT-type models. Unexpectedly, random pruning results in performance that is comparable to the best established methods, across multiple natural l…
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
Multimedia Laboratory (MMLab), The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Salesforce AI Research（Salesforce 人工智能研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；CPII under InnoHK（InnoHK 创新香港研发平台旗下产品可靠性及系统安全研发中心）
LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models
School of Microelectronics, Southern University of Science and Technology（微电子学院，南方科技大学）；Department of Electrical and Electronic Engineering, University of Hong Kong（电气与电子工程系，香港大学）
The success of convolutional neural networks (CNNs) in computer vision applications has been accompanied by a significant increase of computation and memory costs, which prohibits its usage on resource-limited environments such as mobile or embedded devices. To this end, the research of CNN compression has recently become emerging. In this paper, we propose a novel filter pruning scheme, termed structured sparsity regularization (SSR), to simultaneously speedup the computation and reduce the memory overhead of CNNs, which can be well supported by various off-the-shelf deep learning libraries. …
Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models
Edison Academy Magnet School（爱迪生学院磁石学校）；Massachusetts Institute of Technology（麻省理工学院）；State University of New York College at Plattsburgh（纽约州立大学普拉茨堡学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Independent Researcher（独立研究员）
Debias-SparseGPT: Bias-Aware Pruning for Large Language Models
Laboratoire Hubert Curien（于贝尔·屈里安实验室）；Université Claude Bernard Lyon 1（里昂第一大学（克洛德·贝尔纳））；Université Lumière Lyon 2（里昂第二大学（吕米埃尔））；École Centrale de Lyon（里昂中央理工学院）
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）
Language model debiasing has emerged as an important field of study in the NLP community. Numerous debiasing techniques were proposed, but bias ablation remains an unaddressed issue. We demonstrate a novel framework for inspecting bias in pre-trained transformer-based language models via movement pruning. Given a model and a debiasing objective, our framework finds a subset of the model containing less bias than the original model. We implement our framework by pruning the model while fine-tuning it on the debiasing objective. Optimized are only the pruning scores - parameters coupled with the…
Reducing computation cost, inference latency, and memory footprint of neural networks are frequently cited as research motivations for pruning and sparsity. However, operationalizing those benefits and understanding the end-to-end effect of algorithm design and regularization on the runtime execution is not often examined in depth. Here we apply structured and unstructured pruning to attention weights of transformer blocks of the BERT language model, while also expanding block sparse representation (BSR) operations in the TVM compiler. Integration of BSR operations enables the TVM runtime exec…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；National Institute of Informatics（日本国立信息学研究所）
Are end-to-end text-to-speech (TTS) models over-parametrized? To what extent can these models be pruned, and what happens to their synthesis capabilities? This work serves as a starting point to explore pruning both spectrogram prediction networks and vocoders. We thoroughly investigate the tradeoffs between sparsity and its subsequent effects on synthetic speech. Additionally, we explored several aspects of TTS pruning: amount of finetuning data versus sparsity, TTS-Augmentation to utilize unspoken text, and combining knowledge distillation and pruning. Our findings suggest that not only are …
