This paper presents Thanos, a novel weight-pruning algorithm designed to reduce the memory footprint and enhance the computational efficiency of large language models (LLMs) by removing redundant weights while maintaining accuracy. Thanos introduces a block-wise pruning strategy with adaptive masks that dynamically adjust to weight importance, enabling flexible sparsity patterns and structured formats, such as $n:m$ sparsity, optimized for hardware acceleration. Experimental evaluations demonstrate that Thanos achieves state-of-the-art performance in structured pruning and outperforms existing methods in unstructured pruning. By providing an efficient and adaptable approach to model compression, Thanos offers a practical solution for deploying large models in resource-constrained environments.
Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning…
MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；City University of Hong Kong（香港城市大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）
CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机学院）；Cognitive AI Lab, Shanghai Huawei Technologies, China（上海华为技术有限公司认知人工智能实验室）
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
The University of Hong Kong（香港大学）；OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室开放智能研究院）；Simon Fraser University（西蒙菲莎大学）；Southern University of Science and Technology（南方科技大学）
Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models
Shenzhen Future Network of Intelligence Institute（深圳未来网络智能研究院）；Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来网络智能重点实验室）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Beijing Normal University（北京师范大学）；Alibaba Group（阿里巴巴集团）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）
The sizes of pretrained language models make them challenging and expensive to use when there are multiple desired downstream tasks. In this work, we adopt recent strategies for model pruning during finetuning to explore the question of whether it is possible to prune a single encoder so that it can be used for multiple tasks. We allocate a fixed parameter budget and compare pruning a single model with a multitask objective against the best ensemble of single-task models. We find that under two pruning strategies (element-wise and rank pruning), the approach with the multitask objective outper…
STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models
Peking University（北京大学）；Nanyang Technological University, Singapore（南洋理工大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；De Artificial Intelligence Lab（De人工智能实验室）
Pre-training and then fine-tuning large language models is commonly used to achieve state-of-the-art performance in natural language processing (NLP) tasks. However, most pre-trained models suffer from low inference speed. Deploying such large models to applications with latency constraints is challenging. In this work, we focus on accelerating the inference via conditional computations. To achieve this, we propose a novel idea, Magic Pyramid (MP), to reduce both width-wise and depth-wise computation via token pruning and early exiting for Transformer-based models, particularly BERT. The forme…
Efficient Post-Training Pruning of Large Language Models with Statistical Correction
Department of Computer Science and Engineering, Santa Clara University, Santa Clara, CA, USA（计算机科学与工程系，圣克拉拉大学）；Futurewei Technologies, Inc., USA（未来韦艾技术公司）
IG-Pruning: Input-Guided Block Pruning for Large Language Models
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）；Key Laboratory of AI Safety, Chinese Academy of Sciences（人工智能安全重点实验室，中国科学院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
SparseSwaps: Tractable LLM Pruning Mask Refinement at Scale
Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany（人工智能与社会、科学与技术部门，柏林Zuse研究所，德国）；Institute of Mathematics, Technische Universität Berlin, Germany（数学研究所，柏林技术大学，德国）
IntraSlice: Towards High-Performance Structural Pruning with Block-Intra PCA for LLMs
Nanjing University of Science and Technology（南京理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing National Research Center for Information Science and Technology（北京信息科学研究院）
Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models
College of Computing and Data Science（计算与数据科学学院）；Nanyang Technological University（南洋理工大学）；School of Software（软件学院）；Shandong University（山东大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；The University of British Columbia（不列颠哥伦比亚大学）
Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions
School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院，南京，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用重点实验室（东南大学），教育部，中国）
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences（中国科学院智能信息处理重点实验室）；Zhejiang University（浙江大学）；Computer Center of Peking University（北京大学计算中心）；Peng Cheng Laboratory（鹏城实验室）
The proposed pruning strategy offers merits over weight-based pruning techniques: (1) it avoids irregular memory access since representations and matrices can be squeezed into their smaller but dense counterparts, leading to greater speedup; (2) in a manner of top-down pruning, the proposed method operates from a more global perspective based on training signals in the top layer, and prunes each layer by propagating the effect of global signals through layers, leading to better performances at the same sparsity level. Extensive experiments show that at the same sparsity level, the proposed str…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Southern California（南加州大学）
Many efforts have been made to facilitate natural language processing tasks with pre-trained language models (LMs), and brought significant improvements to various applications. To fully leverage the nearly unlimited corpora and capture linguistic information of multifarious levels, large-size LMs are required; but for a specific task, only parts of these information are useful. Such large-sized LMs, even in the inference stage, may cause heavy computation workloads, making them too time-consuming for large-scale applications. Here we propose to compress bulky LMs while preserving useful infor…
Tsinghua University（清华大学）；Huawei Wireless Technology Lab（华为无线技术实验室）；Hong Kong University of Science and Technology（香港科技大学）
Federated learning (FL) has prevailed as an efficient and privacy-preserved scheme for distributed learning. In this work, we mainly focus on the optimization of computation and communication in FL from a view of pruning. By adopting layer-wise pruning in local training and federated updating, we formulate an explicit FL pruning framework, FedLP (Federated Layer-wise Pruning), which is model-agnostic and universal for different types of deep learning models. Two specific schemes of FedLP are designed for scenarios with homogeneous local models and heterogeneous ones. Both theoretical and exper…
We present a post-training weight pruning method for deep neural networks that achieves accuracy levels tolerable for the production setting and that is sufficiently fast to be run on commodity hardware such as desktop CPUs or edge devices. We propose a data-free extension of the approach for computer vision models based on automatically-generated synthetic fractal images. We obtain state-of-the-art results for data-free neural network pruning, with ~1.5% top@1 accuracy drop for a ResNet50 on ImageNet at 50% sparsity rate. When using real data, we are able to get a ResNet50 model on ImageNet w…
