LiePrune: Lie Group and Quantum Geometric Dual Representation for One-Shot Structured Pruning of Quantum Neural Networks
School of Computer, Jiangsu University of Science and Technology（江苏科技大学计算机学院）；Department of Electrical and Computer Engineering, University of Nevada, Las Vegas（内华达大学拉斯维加斯分校电子与计算机工程系）
Quantum neural networks (QNNs) and parameterized quantum circuits (PQCs) are key building blocks for near-term quantum machine learning. However, their scalability is constrained by excessive parameters, barren plateaus, and hardware limitations. We propose LiePrune, the first mathematically grounded one-shot structured pruning framework for QNNs that leverages Lie group structure and quantum geometric information. Each gate is jointly represented in a Lie group--Lie algebra dual space and a quantum geometric feature space, enabling principled redundancy detection and aggressive compression. Experiments on quantum classification (MNIST, FashionMNIST), quantum generative modeling (Bars-and-Stripes), and quantum chemistry (LiH VQE) show that LiePrune achieves over $10\times$ compression with negligible or even improved task performance, while providing provable guarantees on redundancy detection, functional approximation, and computational complexity.
Ritsumeikan University（立命馆大学）；Hitotsubashi University（一桥大学）；Nagoya University（名古屋大学）；Dalian University of Technology（大连理工大学）
One-shot Network Pruning at Initialization (OPaI) is an effective method to decrease network pruning costs. Recently, there is a growing belief that data is unnecessary in OPaI. However, we obtain an opposite conclusion by ablation experiments in two representative OPaI methods, SNIP and GraSP. Specifically, we find that informative data is crucial to enhancing pruning performance. In this paper, we propose two novel methods, Discriminative One-shot Network Pruning (DOP) and Super Stitching, to prune the network by high-level visual discriminative image patches. Our contributions are as follow…
Osaka University（大阪大学）；University of Electronic Science and Technology of China（电子科技大学）
Recently, tile pruning has been widely studied to accelerate the inference of deep neural networks (DNNs). However, we found that the loss due to tile pruning, which can eliminate important elements together with unimportant elements, is large on trained DNNs. In this study, we propose a one-shot reparameterization method, called TileTrans, to reduce the loss of tile pruning. Specifically, we repermute the rows or columns of the weight matrix such that the model architecture can be kept unchanged after reparameterization. This repermutation realizes the reparameterization of the DNN model with…
Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation
Apurba-NSU R&D Lab, Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh（阿普尔巴-NSU研发实验室，电气与计算机工程系，北方南大学，达卡，孟加拉国）；Apurba Technologies, Sunnyvale, CA 94085, USA（阿普尔巴技术公司，硅谷，加利福尼亚州94085，美国）
Institute for Infocomm Research, Agency for Science, Technology and Research（新加坡科技研究局资讯通信研究院）；College of Computer Science, Sichuan University（四川大学计算机学院）；Nanyang Technological University（南洋理工大学）
As Deep Neural Networks (DNNs) usually are overparameterized and have millions of weight parameters, it is challenging to deploy these large DNN models on resource-constrained hardware platforms, e.g., smartphones. Numerous network compression methods such as pruning and quantization are proposed to reduce the model size significantly, of which the key is to find suitable compression allocation (e.g., pruning sparsity and quantization codebook) of each layer. Existing solutions obtain the compression allocation in an iterative/manual fashion while finetuning the compressed model, thus sufferin…
As language models have grown in parameters and layers, it has become much harder to train and infer with them on single GPUs. This is severely restricting the availability of large language models such as GPT-3, BERT-Large, and many others. A common technique to solve this problem is pruning the network architecture by removing transformer heads, fully-connected weights, and other modules. The main challenge is to discern the important parameters from the less important ones. Our goal is to find strong metrics for identifying such parameters. We thus propose two strategies: Cam-Cut based on t…
CUP: Cluster Pruning for Compressing Deep Neural Networks
Georgia Institute of Technology（佐治亚理工学院）；IQVIA（艾昆纬）
我们提出了集群修剪（CUP）来压缩和加速深度神经网络。我们的方法通过基于来自传入和传出权重连接的特征对其进行聚类来修剪类似的过滤器。利用CUP，我们克服了先前工作的两个限制-（1）非均匀修剪：CUP可以有效地确定神经网络的每一层中要修剪的过滤器的理想数量。这与现有方法形成对比，现有方法均匀地修剪所有层或以其他方式使用资源密集型方法，例如手动灵敏度分析或强化学习来确定理想数量。(2)单次操作：我们将CUP扩展到CUP-SS（用于CUP Single Shot），从而将修剪集成到初始训练阶段本身。与传统的修剪管道相比，这可以节省大量的训练时间。通过对多个数据集（MNIST，CIFAR-10和Imagenet）和模型（VGG-16，Resnets-18/34/56）的广泛评估，我们表明CUP优于最新技术水平。具体来说，CUP-SS在Imagenet上训练的Resnet-50模型实现了2.2倍的浮点数减少，同时保持在0.9%的前5名准确度内。与原始Resnet-50相比，它节省了超过14小时的培训时间。代码再现结果是可用的。
Tsinghua University（清华大学）；Nuctech AI R&D Center（Nuctech人工智能研发中心）
Filter pruning has drawn more attention since resource constrained platform requires more compact model for deployment. However, current pruning methods suffer either from the inferior performance of one-shot methods, or the expensive time cost of iterative training methods. In this paper, we propose a balanced filter pruning method for both performance and pruning speed. Based on the filter importance criteria, our method is able to prune a layer with approximate layer-wise optimal pruning rate at preset loss variation. The network is pruned in the layer-wise way without the time consuming pr…
Effective pruning of web-scale datasets based on complexity of concept clusters
University of Tübingen（蒂宾根大学）；Meta AI (FAIR)（Meta AI（FAIR））；ELLIS Institute Tübingen；Max-Planck Institute for Intelligent Systems Tübingen（蒂宾根马克斯·普朗克智能系统研究所）；University of California San Diego（加州大学圣迭戈分校）；DatologyAI
Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
Stanford University（斯坦福大学）；North China Electric Power University（华北电力大学）；University of Technology Sydney（悉尼科技大学）；Harvard University（哈佛大学）；University of Science and Technology of China（中国科学技术大学）
MIT EECS（麻省理工学院电气工程与计算机科学系）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
Magnitude pruning is a common, effective technique to identify sparse subnetworks at little cost to accuracy. In this work, we ask whether a particular architecture's accuracy-sparsity tradeoff can be improved by combining pruning information across multiple runs of training. From a shared ResNet-20 initialization, we train several network copies (\emph{siblings}) to completion using different SGD data orders on CIFAR-10. While the siblings' pruning masks are naively not much more similar than chance, starting sibling training after a few epochs of shared pretraining significantly increases pr…
Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. While this can be elegantly achieved by applying sparsity-inducing regularisation on BatchNorm parameters, an L1 penalty would shrink all scaling factors rather than just those of superfluous neurons. To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance. Under our method, the weights of unnecessary neurons effectively recede, producing a polarised bimo…
Robert Bosch GmbH（罗伯特·博世有限公司）；Lübeck University（吕贝克大学）
State-of-the-art deep neural network (DNN) pruning techniques, applied one-shot before training starts, evaluate sparse architectures with the help of a single criterion -- called pruning score. Pruning weights based on a solitary score works well for some architectures and pruning rates but may also fail for other ones. As a common baseline for pruning scores, we introduce the notion of a generalized synaptic score (GSS). In this work we do not concentrate on a single pruning criterion, but provide a framework for combining arbitrary GSSs to create more powerful pruning strategies. These COmb…
Introducing sparsity in a neural network has been an efficient way to reduce its complexity while keeping its performance almost intact. Most of the time, sparsity is introduced using a three-stage pipeline: 1) train the model to convergence, 2) prune the model according to some criterion, 3) fine-tune the pruned model to recover performance. The last two steps are often performed iteratively, leading to reasonable results but also to a time-consuming and complex process. In our work, we propose to get rid of the first step of the pipeline and to combine the two other steps in a single pruning…
Eindhoven University of Technology（埃因霍温理工大学）；University of Texas at Austin（得克萨斯大学奥斯汀分校）；University of Twente（特文特大学）；University of Leeds（利兹大学）；JD Explore Academy（京东探索研究院）；University of Jyväskylä（于韦斯屈莱大学）
Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization). The former method suffers from an extremely large computation cost and the latter usually struggles with insufficient performance. In comparison, during-training pruning, a class of pruning methods that simultaneously enjoys the training/inference efficiency and the comparable performance, temporarily, has been less explored. To better understand during-training p…
Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application to a new scenario where the regularization grows large gradually to tackle two central problems of pruning: pruning schedule and weight importance scoring. (1) The former topic is newly brought up in this work, which we find critical to the pruning performance while receives little research attention. Specifically, we propose an L2 regularization variant with rising penalty factors and show it …
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Filter pruning has attracted increasing attention in recent years for its capacity in compressing and accelerating convolutional neural networks. Various data-independent criteria, including norm-based and relationship-based ones, were proposed to prune the most unimportant filters. However, these state-of-the-art criteria fail to fully consider the dissimilarity of filters, and thus might lead to performance degradation. In this paper, we first analyze the limitation of relationship-based criteria with examples, and then introduce a new data-independent criterion, Weighted Hybrid Criterion (W…
Slimmable Neural Networks (S-Net) is a novel network which enabled to select one of the predefined proportions of channels (sub-network) dynamically depending on the current computational resource availability. The accuracy of each sub-network on S-Net, however, is inferior to that of individually trained networks of the same size due to its difficulty of simultaneous optimization on different sub-networks. In this paper, we propose Slimmable Pruned Neural Networks (SP-Net), which has sub-network structures learned by pruning instead of adopting structures with the same proportion of channels …
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Landskape AI（Landskape人工智能）；UdeM（蒙特利尔大学）；IIT-Delhi（印度理工学院德里分校）；UT-Austin（德克萨斯大学奥斯汀分校）
With the latest advances in deep learning, there has been a lot of focus on the online learning paradigm due to its relevance in practical settings. Although many methods have been investigated for optimal learning settings in scenarios where the data stream is continuous over time, sparse networks training in such settings have often been overlooked. In this paper, we explore the problem of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA). We propose a novel way of progressive pruning, referred to as \…
Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM
Sensetime Research（商汤科技研究院）；Tsinghua University（清华大学）；Syracuse University（雪城大学）；Northeastern University（东北大学）；Florida International University（佛罗里达国际大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
权值修剪和权值量化是DNN模型压缩的两个重要方面。这些技术的先前工作主要基于化学。最近的一项工作使用先进的优化技术ADMM（Alternating Direction Methods of Multipliers）开发了DNN权重修剪的系统框架，实现了最先进的权重修剪结果之一。在这项工作中，我们首先扩展这样的一杆ADMM为基础的框架，以保证解决方案的可行性，并提供快速收敛速度，以及推广到权重量化。我们进一步开发了一个多步骤的渐进式DNN权重修剪和量化框架，具有双重好处：（i）由于ADMM正则化的特殊属性，实现了进一步的权重修剪/量化，以及（ii）减少了每个步骤中的搜索空间。大量的实验结果表明，与以前的工作相比，优越的性能。一些亮点：（i）我们分别在LeNet-5、AlexNet和ResNet-50模型上实现了246倍、36倍和8倍的权重修剪，（几乎）零精度损失;（ii）在AlexNet中甚至实现了显著的61倍权重修剪（ImageNet）与先前的工作相比，实际准确性仅略有下降;（iii）我们是第一批为ResNet和MobileNet模型得出显着权重修剪结果的人之一;（iv）我们为MNIST和CIFAR-10分别导出了第一个无损的、完全二进制化的（所有层）LeNet-5和VGG-16;以及（v）我们为ImageNet导出了第一个完全二进制化的（所有层）ResNet，具有合理的准确性损失…
Quantum-Inspired Spectral Geometry for Neural Operator Equivalence and Structured Pruning
School of Computer Jiangsu University of Science and Technology（江苏科技大学计算机学院）；School of Computer and Software Nanjing University of Information Science and Technology（南京信息工程大学计算机与软件学院）；Department of Electrical and Computer Engineering University of Nevada Las Vegas（内华达大学拉斯维加斯分校电气与计算机工程系）
Department of Machine Learning and Neural Computing（机器学习与神经计算系）；Donders Institute for Brain, Cognition and Behaviour（大脑、认知与行为研究所）；Radboud University（拉德堡德大学）
Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers
Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因茨·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））
Scalable iterative pruning of large language and vision models using block coordinate descent
Amazon Quantum Solutions Lab（亚马逊量子解决方案实验室）；AWS Center for Quantum Computing（AWS量子计算中心）；Fidelity Center for Applied Technology（富达应用技术中心）；AI Center of Excellence, FMR LLC（FMR LLC卓越人工智能中心）
