This paper provides a fresh view of the neural network (NN) pruning problem through the lens of graph theory. To achieve effective pruning, we aim to identify the main NN data flows and the corresponding NN connections that are most and least important for the performance of the full model. Unlike the standard approach to NN data flow analysis, which is based on information theory, we employ the notion of graph curvature, specifically Ollivier-Ricci curvature (ORC). ORC has been successfully used to identify important graph edges in various domains such as road traffic analysis, biological networks, and social networks. In particular, edges with negative ORC are considered bottlenecks and are therefore critical to the graph's overall connectivity, whereas positive-ORC edges are less essential. We use this intuition for NNs to (1) construct a graph induced by the NN structure and introduce the notion of neural curvature (NC) based on ORC; (2) calculate curvatures based on activation patterns for a set of input examples; and (3) demonstrate that NC can be used to rank edges according to their importance for overall NN functionality. We evaluate our method through pruning experiments on a variety of small and medium size models trained on three image datasets: MNIST, CIFAR-10, and CIFAR-100. The results indicate that our method can identify a larger number of unimportant edges compared to existing pruning methods.
P$^2$ Law: Scaling Law for Post-Training After Model Pruning
Engineering Research Center of Database and Business Intelligence, MOE, China（数据库与商业智能工程研究中心，教育部，中国）；School of Information, Renmin University of China,Beijing, China（信息学院，中国人民大学，北京，中国）；Key Laboratory of Data Engineering and Knowledge Engineering, MOE, China（数据工程与知识工程重点实验室，教育部，中国）；Zhipu AI, China（智谱AI，中国）
We present a post-training weight pruning method for deep neural networks that achieves accuracy levels tolerable for the production setting and that is sufficiently fast to be run on commodity hardware such as desktop CPUs or edge devices. We propose a data-free extension of the approach for computer vision models based on automatically-generated synthetic fractal images. We obtain state-of-the-art results for data-free neural network pruning, with ~1.5% top@1 accuracy drop for a ResNet50 on ImageNet at 50% sparsity rate. When using real data, we are able to get a ResNet50 model on ImageNet w…
From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models
Institute of Digital Twin, Eastern Institute of Technology（数字孪生研究院，东部技术研究所）；Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；Tencent Hunyuan / AI Lab（腾讯文英/人工智能实验室）；Université Paris Dauphine - PSL（巴黎-萨克勒大学）
Pruning enables appealing reductions in network memory footprint and time complexity. Conventional post-training pruning techniques lean towards efficient inference while overlooking the heavy computation for training. Recent exploration of pre-training pruning at initialization hints on training cost reduction via pruning, but suffers noticeable performance degradation. We attempt to combine the benefits of both directions and propose a policy that prunes as early as possible during training without hurting performance. Instead of pruning at initialization, our method exploits initial dense t…
Compact Bayesian Neural Networks via pruned MCMC sampling
ITTC ARC Centre for Data Analytics for Resources and Environment（ITTC ARC资源与环境数据分析中心）；University of Sydney（悉尼大学）；UNSW Data Science Hub（新南威尔士大学数据科学中心）；University of New South Wales（新南威尔士大学）；Geocoastal Research Group（地理海岸研究组）；Transitional Artificial Intelligence Research Group（过渡人工智能研究组）
Eindhoven University of Technology（埃因霍温理工大学）；University of Texas at Austin（得克萨斯大学奥斯汀分校）；University of Twente（特文特大学）；University of Leeds（利兹大学）；JD Explore Academy（京东探索研究院）；University of Jyväskylä（于韦斯屈莱大学）
Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization). The former method suffers from an extremely large computation cost and the latter usually struggles with insufficient performance. In comparison, during-training pruning, a class of pruning methods that simultaneously enjoys the training/inference efficiency and the comparable performance, temporarily, has been less explored. To better understand during-training p…
The goal of Bayesian deep learning is to provide uncertainty quantification via the posterior distribution. However, exact inference over the weight space is computationally intractable due to the ultra-high dimensions of the neural network. Variational inference (VI) is a promising approach, but naive application on weight space does not scale well and often underperform on predictive accuracy. In this paper, we propose a new adaptive variational Bayesian algorithm to train neural networks on weight space that achieves high predictive accuracy. By showing that there is an equivalence to Stoch…
Existing methods of pruning deep neural networks focus on removing unnecessary parameters of the trained network and fine tuning the model afterwards to find a good solution that recovers the initial performance of the trained model. Unlike other works, our method pays special attention to the quality of the solution in the compressed model and inference computation time by pruning neurons. The proposed algorithm directs the parameters of the compressed model toward a flatter solution by exploring the spectral radius of Hessian which results in better generalization on unseen data. Moreover, t…
A novel method to identify salient computational paths within randomly wired neural networks before training is proposed. The computational graph is pruned based on a node mass probability function defined by local graph measures and weighted by hyperparameters produced by a reinforcement learning-based controller neural network. We use the definition of Ricci curvature to remove edges of low importance before mapping the computational graph to a neural network. We show a reduction of almost $35\%$ in the number of floating-point operations (FLOPs) per pass, with no degradation in performance.…
Efficient Post-Training Pruning of Large Language Models with Statistical Correction
Department of Computer Science and Engineering, Santa Clara University, Santa Clara, CA, USA（计算机科学与工程系，圣克拉拉大学）；Futurewei Technologies, Inc., USA（未来韦艾技术公司）
CoCurve: Cross-Module Co-Pruning Curvature for Structured LLM Pruning
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Interdisciplinary Graduate Programme, Nanyang Technological University（南洋理工大学跨学科研究生项目）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）
IST Austria（奥地利科学技术研究所）；ETH Zurich（苏黎世联邦理工学院）；Neural Magic（神经魔法公司）
We consider the problem of model compression for deep neural networks (DNNs) in the challenging one-shot/post-training setting, in which we are given an accurate trained model, and must compress it without any retraining, based only on a small amount of calibration input data. This problem has become popular in view of the emerging software and hardware support for executing models compressed via pruning and/or quantization with speedup, and well-performing solutions have been proposed independently for both compression approaches. In this paper, we introduce a new compression framework which …
Adaptive Signal Resuscitation: Channel-wise Post-Pruning Repair for Sparse Vision Networks
Department of Mathematical and Statistical Sciences, Marquette University（马歇尔大学数学与统计科学系）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Cornell Ann S. Bowers College of Computing and Information Science, Cornell University（康奈尔大学安·S·博尔斯计算与信息科学学院）
Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）
Application of discrete Ricci curvature in pruning randomly wired neural networks: A case study with chest x-ray classification of COVID-19
The Institute of Mathematical Sciences (IMSc), Chennai 600113, India（数学科学研究所）；Institute of Computer Science（计算机科学研究所）；Homi Bhabha National Institute (HBNI), Mumbai 400094, India（霍米·巴哈瓦国家研究所）
Pruning is an effective way to reduce the huge inference cost of Transformer models. However, prior work on pruning Transformers requires retraining the models. This can add high training cost and high complexity to model deployment, making it difficult to use in many practical situations. To address this, we propose a fast post-training pruning framework for Transformers that does not require any retraining. Given a resource constraint and a sample dataset, our framework automatically prunes the Transformer model using structured sparsity methods. To retain high accuracy without retraining, w…
Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retraining
Department of Electrical and Computer Engineering, North South University, Dhaka, 1229, Bangladesh（电气与计算机工程系，北南大学，达卡，孟加拉国）；Data Science and AI, University of Doha for Science and Technology, Doha, Qatar（数据科学与人工智能，多哈科学技术大学，多哈，卡塔尔）
Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches
Applied AI（应用人工智能）；Artificial Intelligence Research Institute（人工智能研究所）；HSE University（俄罗斯高等经济大学）；Sb AI Lab（Sb人工智能实验室）；MWS AI（MWS人工智能）；Yandex；Independent Researcher（独立研究者）
Eötvös Loránd University（厄特沃什·罗兰大学）；Alfréd Rényi Institute of Mathematics（阿尔弗雷德·雷尼数学研究所）
The Lottery Ticket Hypothesis postulates that a freshly initialized neural network contains a small subnetwork that can be trained in isolation to achieve similar performance as the full network. Our paper examines several alternatives to search for such subnetworks. We conclude that incorporating a data dependent component into the pruning criterion in the form of the gradient of the training loss -- as done in the SNIP method -- consistently improves the performance of existing pruning algorithms.
A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
Department for AI in Society, Science, and Technology, Zuse Institute Berlin（人工智能社会、科学与技术系，柏林Zuse研究所）；Institute of Mathematics, Technische Universität Berlin（数学系，柏林技术大学）
ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Washington（华盛顿大学）；Princeton University（普林斯顿大学）；Georgia Institute of Technology（佐治亚理工学院）
Beware of Calibration Data for Pruning Large Language Models
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Key Laboratory of Data Intelligence and Advanced Computing, Soochow University（苏州大学数据智能与先进计算重点实验室）；Huawei Cloud, China（华为云（中国））
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zürich（苏黎世联邦理工学院）；University of Tübingen（蒂宾根大学）
We introduce Dirichlet pruning, a novel post-processing technique to transform a large neural network model into a compressed one. Dirichlet pruning is a form of structured pruning that assigns the Dirichlet distribution over each layer's channels in convolutional layers (or neurons in fully-connected layers) and estimates the parameters of the distribution over these units using variational inference. The learned distribution allows us to remove unimportant units, resulting in a compact architecture containing only crucial features for a task at hand. The number of newly introduced Dirichlet …
