In order to deploy deep convolutional neural networks (CNNs) on resource-limited devices, many model pruning methods for filters and weights have been developed, while only a few to layer pruning. However, compared with filter pruning and weight pruning, the compact model obtained by layer pruning has less inference time and run-time memory usage when the same FLOPs and number of parameters are pruned because of less data moving in memory. In this paper, we propose a simple layer pruning method using fusible residual convolutional block (ResConv), which is implemented by inserting shortcut connection with a trainable information control parameter into a single convolutional layer. Using ResConv structures in training can improve network accuracy and train deep plain networks, and adds no additional computation during inference process because ResConv is fused to be an ordinary convolutional layer after training. For layer pruning, we convert convolutional layers of network into ResConv with a layer scaling factor. In the training process, the L1 regularization is adopted to make the scaling factors sparse, so that unimportant layers are automatically identified and then removed, resulting in a model of layer reduction. Our pruning method achieves excellent performance of compression and acceleration over the state-of-the-arts on different datasets, and needs no retraining in the case of low pruning rate. For example, with ResNet-110, we achieve a 65.5%-FLOPs reduction by removing 55.5% of the parameters, with only a small loss of 0.13% in top-1 accuracy on CIFAR-10.
Filter level pruning is an effective method to accelerate the inference speed of deep CNN models. Although numerous pruning algorithms have been proposed, there are still two open issues. The first problem is how to prune residual connections. We propose to prune both channels inside and outside the residual connections via a KL-divergence based criterion. The second issue is pruning with limited data. We observe an interesting phenomenon: directly pruning on a small dataset is usually worse than fine-tuning a small model which is pruned or trained from scratch on the large dataset. Knowledge …
Improving weight sparsity is a common strategy for producing light-weight deep neural networks. However, pruning models with residual learning is more challenging. In this paper, we introduce Variance-Aware Cross-Layer (VACL), a novel approach to address this problem. VACL consists of two parts, a Cross-Layer grouping and a Variance Aware regularization. In Cross-Layer grouping the $i^{th}$ filters of layers connected by skip-connections are grouped into one regularization group. Then, the Variance-Aware regularization term takes into account both the first and second-order statistics of the c…
Automated Tomato Maturity Estimation Using an Optimized Residual Model with Pruning and Quantization Techniques
Auburn University（奥本大学）；National Yang-Ming Chiao Tung University（国立阳明交通大学）；University of Agriculture, Faisalabad（费萨拉巴德农业大学）；University of Georgia（佐治亚大学）；University of Wyoming（怀俄明大学）；National Institute of Food Science and Technology（国家食品科学与技术研究所）
University of Science and Technology of China（中国科学技术大学）；HeFei University of Technology（合肥工业大学）
Pruning filters is an effective method for accelerating deep neural networks (DNNs), but most existing approaches prune filters on a pre-trained network directly which limits in acceleration. Although each filter has its own effect in DNNs, but if two filters are the same with each other, we could prune one safely. In this paper, we add an extra cluster loss term in the loss function which can force filters in each cluster to be similar online. After training, we keep one filter in each cluster and prune others and fine-tune the pruned network to compensate for the loss. Particularly, the clus…
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）；Engineering Research Center of Safety Critical Industrial Measurement and Control Technology, Ministry of Education, Hefei University of Technology（合肥工业大学安全关键工业测量与控制技术教育部工程研究中心）
To apply deep CNNs to mobile terminals and portable devices, many scholars have recently worked on the compressing and accelerating deep convolutional neural networks. Based on this, we propose a novel uniform channel pruning (UCP) method to prune deep CNN, and the modified squeeze-and-excitation blocks (MSEB) is used to measure the importance of the channels in the convolutional layers. The unimportant channels, including convolutional kernels related to them, are pruned directly, which greatly reduces the storage cost and the number of calculations. There are two types of residual blocks in …
Convolutional neural networks (CNNs) are able to attain better visual recognition performance than fully connected neural networks despite having much fewer parameters due to their parameter sharing principle. Modern architectures usually contain a small number of fully-connected layers, often at the end, after multiple layers of convolutions. In some cases, most of the convolutions can be eliminated without suffering any loss in recognition performance. However, there is no solid recipe to detect the hidden subset of convolutional neurons that is responsible for the majority of the recognitio…
We introduce Bayesian Bits, a practical method for joint mixed precision quantization and pruning through gradient based optimization. Bayesian Bits employs a novel decomposition of the quantization operation, which sequentially considers doubling the bit width. At each new bit width, the residual error between the full precision value and the previously rounded value is quantized. We then decide whether or not to add this quantized residual error for a higher effective bit width and lower quantization noise. By starting with a power-of-two bit width, this decomposition will always produce har…
Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency
Nanjing University of Science and Technology（南京理工大学）；State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery（先进工程机械智能制造国家重点实验室）；Zhejiang University（浙江大学）；Nanjing Normal University（南京师范大学）
Slimmable Neural Networks (S-Net) is a novel network which enabled to select one of the predefined proportions of channels (sub-network) dynamically depending on the current computational resource availability. The accuracy of each sub-network on S-Net, however, is inferior to that of individually trained networks of the same size due to its difficulty of simultaneous optimization on different sub-networks. In this paper, we propose Slimmable Pruned Neural Networks (SP-Net), which has sub-network structures learned by pruning instead of adopting structures with the same proportion of channels …
Lahore University of Management Sciences（拉合尔管理科学大学）
One of the major challenges in deploying deep neural network architectures is their size which has an adverse effect on their inference time and memory requirements. Deep CNNs can either be pruned width-wise by removing filters based on their importance or depth-wise by removing layers and blocks. Width wise pruning (filter pruning) is commonly performed via learnable gates or switches and sparsity regularizers whereas pruning of layers has so far been performed arbitrarily by manually designing a smaller network usually referred to as a student network. We propose a comprehensive pruning stra…
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task.…
Residual Networks with convolutional layers are widely used in the field of machine learning. Since they effectively extract features from input data by stacking multiple layers, they can achieve high accuracy in many applications. However, the stacking of many layers raises their computation costs. To address this problem, we propose Network Implosion, it erases multiple layers from Residual Networks without degrading accuracy. Our key idea is to introduce a priority term that identifies the importance of a layer; we can select unimportant layers according to the priority and erase them after…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械与车辆学院）；University of Chinese Academy of Sciences（中国科学院大学）；College of Engineering and Computer Science, Australian National University（澳大利亚国立大学工程与计算机科学学院）
Currently, an increasing number of model pruning methods are proposed to resolve the contradictions between the computer powers required by the deep learning models and the resource-constrained devices. However, most of the traditional rule-based network pruning methods can not reach a sufficient compression ratio with low accuracy loss and are time-consuming as well as laborious. In this paper, we propose Automatic Block-wise and Channel-wise Network Pruning (ABCP) to jointly search the block-wise and channel-wise pruning action with deep reinforcement learning. A joint sample algorithm is pr…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Jilin University（吉林大学）；University of New South Wales（新南威尔士大学）
The increasing computational cost of deep neural network models limits the applicability of intelligent applications on resource-constrained edge devices. While a number of neural network pruning methods have been proposed to compress the models, prevailing approaches focus only on parametric operators (e.g., convolution), which may miss optimization opportunities. In this paper, we present a novel fusion-catalyzed pruning approach, called FuPruner, which simultaneously optimizes the parametric and non-parametric operators for accelerating neural networks. We introduce an aggressive fusion met…
Real-time video segmentation is a crucial task for many real-world applications such as autonomous driving and robot control. Since state-of-the-art semantic segmentation models are often too heavy for real-time applications despite their impressive performance, researchers have proposed lightweight architectures with speed-accuracy trade-offs, achieving real-time speed at the expense of reduced accuracy. In this paper, we propose a novel framework to speed up any architecture with skip-connections for real-time vision tasks by exploiting the temporal locality in videos. Specifically, at the a…
Training-Free Interaction-Aligned Visual Token Pruning for Efficient Embodied Manipulation
Hong Kong University of Science and Technology（香港科学与技术大学）；South China Normal University（华南师范大学）；The Chinese University of Hong Kong（香港中文大学）；University of Science and Technology Beijing（北京科技大学）；National University of Defense Technology（国防科技大学）
Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning
Department of Computing Science University of Aberdeen, UK（计算科学系阿伯丁大学，英国）；Department of Computing Science & Interdisciplinary Institute University of Aberdeen, UK（计算科学系与跨学科研究所阿伯丁大学，英国）；Department of Computer Science University of Exeter, UK（计算机科学系埃克塞特大学，英国）；Intel Labs, USA（英特尔实验室，美国）；Department of Computer Science University of North Carolina at Chapel Hill, US（计算机科学系北卡罗来纳大学教堂山分校，美国）；School of Computer Science and Electronic Engineering University of Surrey, UK（计算机科学与电子工程学院 Surrey大学，英国）
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）
We push forward neural network compression research by exploiting a novel challenging task of large-scale conditional generative adversarial networks (GANs) compression. To this end, we propose a gradually shrinking GAN (PPCD-GAN) by introducing progressive pruning residual block (PP-Res) and class-aware distillation. The PP-Res is an extension of the conventional residual block where each convolutional layer is followed by a learnable mask layer to progressively prune network parameters as training proceeds. The class-aware distillation, on the other hand, enhances the stability of training b…
Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）；ETH Zürich（苏黎世联邦理工学院）；ByteDance Inc（字节跳动公司）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
The high computational costs of video super-resolution (VSR) models hinder their deployment on resource-limited devices, (e.g., smartphones and drones). Existing VSR models contain considerable redundant filters, which drag down the inference efficiency. To prune these unimportant filters, we develop a structured pruning scheme called Structured Sparsity Learning (SSL) according to the properties of VSR. In SSL, we design pruning schemes for several key components in VSR models, including residual blocks, recurrent networks, and upsampling networks. Specifically, we develop a Residual Sparsity…
Currently, owing to the ubiquity of mobile devices, online handwritten Chinese character recognition (HCCR) has become one of the suitable choice for feeding input to cell phones and tablet devices. Over the past few years, larger and deeper convolutional neural networks (CNNs) have extensively been employed for improving character recognition performance. However, its substantial storage requirement is a significant obstacle in deploying such networks into portable electronic devices. To circumvent this problem, we propose a novel technique called DropWeight for pruning redundant connections …
University of Freiburg（弗莱堡大学）；Department of Computer Science（计算机科学系）
We analyze the input-output behavior of residual networks from a dynamical system point of view by disentangling the residual dynamics from the output activities before the classification stage. For a network with simple skip connections between every successive layer, and for logistic activation function, and shared weights between layers, we show analytically that there is a cooperation and competition dynamics between residuals corresponding to each input dimension. Interpreting these kind of networks as nonlinear filters, the steady state value of the residuals in the case of attractor net…
ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Carizon；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；University College London（伦敦大学学院）；Qing Yuan Research Institute（清源研究院）
Network compression has been widely studied since it is able to reduce the memory and computation cost during inference. However, previous methods seldom deal with complicated structures like residual connections, group/depth-wise convolution and feature pyramid network, where channels of multiple layers are coupled and need to be pruned simultaneously. In this paper, we present a general channel pruning approach that can be applied to various complicated structures. Particularly, we propose a layer grouping algorithm to find coupled channels automatically. Then we derive a unified metric base…
SynA-ResNet: Spike-driven ResNet Achieved through OR Residual Connection
School of Software, Liaoning Technical University（辽宁工程技术大学软件学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of Electronic Science and Technology of China（电子科技大学）
Factorized layers--operations parameterized by products of two or more matrices--occur in a variety of deep learning contexts, including compressed model training, certain types of knowledge distillation, and multi-head self-attention architectures. We study how to initialize and regularize deep nets containing such layers, examining two simple, understudied schemes, spectral initialization and Frobenius decay, for improving their performance. The guiding insight is to design optimization routines for these networks that are as close as possible to that of their well-tuned, non-decomposed coun…
Most of today's popular deep architectures are hand-engineered to be generalists. However, this design procedure usually leads to massive redundant, useless, or even harmful features for specific tasks. Unnecessarily high complexities render deep nets impractical for many real-world applications, especially those without powerful GPU support. In this paper, we attempt to derive task-dependent compact models from a deep discriminant analysis perspective. We propose an iterative and proactive approach for classification tasks which alternates between (1) a pushing step, with an objective to simu…
Tencent Youtu Lab（腾讯优图实验室）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
Although residual connection enables training very deep neural networks, it is not friendly for online inference due to its multi-branch topology. This encourages many researchers to work on designing DNNs without residual connections at inference. For example, RepVGG re-parameterizes multi-branch topology to a VGG-like (single-branch) model when deploying, showing great performance when the network is relatively shallow. However, RepVGG can not transform ResNet to VGG equivalently because re-parameterizing methods can only be applied to linear blocks and the non-linear layers (ReLU) have to b…
Computer Engineering Department, King Fahd University of Petroleum and Minerals (KFUPM)（计算机工程系，国王法赫德石油与矿物大学）；SDAIA-KFUPM Joint Research Center on Artificial Intelligence, IRC for Intelligent Secure Systems and Department of Computer Engineering, KFUPM（SDAIA-KFUPM人工智能联合研究中心，智能安全系统研究院及计算机工程系，KFUPM）；Center for Intelligent Secure Systems and the Department of Computer Engineering, King Fahd University of Petroleum & Minerals（智能安全系统中心及计算机工程系，国王法赫德石油与矿物大学）；Research Chair of Pervasive and Mobile Computing, Department of Software Engineering, College of Computer and Information Sciences, King Saud University（普及与移动计算研究主席，软件工程系，计算机与信息科学学院，国王沙特大学）
本文提出TinyEcoWeedNet，通过结构化剪枝、量化感知训练和TensorRT加速，在Jetson Orin Nano上实现184 FPS实时杂草检测，模型缩小68.5%，并在CottonWeedDet12上以85.9% mAP50超越YOLO11n和YOLO12n。
Toward a Robust R2D2 Paradigm for Radio-interferometric Imaging: Revisiting Deep Neural Network Training and Architecture
Institute of Sensors, Signals and Systems, Heriot-Watt University, Edinburgh EH14 4AS, United Kingdom（传感器、信号与系统研究所，赫里奥特-瓦特大学）；EPCC, University of Edinburgh, Potterrow, Edinburgh EH8 9BT, United Kingdom（爱丁堡大学埃平中心）
