Model compression and acceleration are attracting increasing attentions due to the demand for embedded devices and mobile applications. Research on efficient convolutional neural networks (CNNs) aims at removing feature redundancy by decomposing or optimizing the convolutional calculation. In this work, feature redundancy is assumed to exist among channels in CNN architectures, which provides some leeway to boost calculation efficiency. Aiming at channel compression, a novel convolutional construction named compact convolution is proposed to embrace the progress in spatial convolution, channel grouping and pooling operation. Specifically, the depth-wise separable convolution and the point-wise interchannel operation are utilized to efficiently extract features. Different from the existing channel compression method which usually introduces considerable learnable weights, the proposed compact convolution can reduce feature redundancy with no extra parameters. With the point-wise interchannel operation, compact convolutions implicitly squeeze the channel dimension of feature maps. To explore the rules on reducing channel redundancy in neural networks, the comparison is made among different point-wise interchannel operations. Moreover, compact convolutions are extended to tackle with multiple tasks, such as acoustic scene classification, sound event detection and image classification. The extensive experiments demonstrate that our compact convolution not only exhibits high effectiveness in several multimedia tasks, but also can be efficiently implemented by benefiting from parallel computation.
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）；Griffith University（格里菲斯大学）
Large-scale graphs are ubiquitous in real-world scenarios and can be trained by Graph Neural Networks (GNNs) to generate representation for downstream tasks. Given the abundant information and complex topology of a large-scale graph, we argue that redundancy exists in such graphs and will degrade the training efficiency. Unfortunately, the model scalability severely restricts the efficiency of training large-scale graphs via vanilla GNNs. Despite recent advances in sampling-based training methods, sampling-based GNNs generally overlook the redundancy issue. It still takes intolerable time to t…
The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；National University of Singapore（新加坡国立大学）
Locality-Aware Redundancy Pruning for LLM Depth Compression
University of Southern California（美国南加州大学）；Neural Superintelligence Lab, MODULABS（MODULABS神经超级智能实验室）；Seoul National University（首尔国立大学）；Inha University（釜山大学）
Rethinking Parameter Sharing as Graph Coloring for Structured Compression
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
Lossy compression algorithms are typically designed and analyzed through the lens of Shannon's rate-distortion theory, where the goal is to achieve the lowest possible distortion (e.g., low MSE or high SSIM) at any given bit rate. However, in recent years, it has become increasingly accepted that "low distortion" is not a synonym for "high perceptual quality", and in fact optimization of one often comes at the expense of the other. In light of this understanding, it is natural to seek for a generalization of rate-distortion theory which takes perceptual quality into account. In this paper, we …
Deploying deep learning models, comprising of non-linear combination of millions, even billions, of parameters is challenging given the memory, power and compute constraints of the real world. This situation has led to research into model compression techniques most of which rely on suboptimal heuristics and do not consider the parameter redundancies due to linear dependence between neuron activations in overparametrized networks. In this paper, we propose a novel model compression approach based on exploitation of linear dependence, that compresses networks by elimination of entire neurons an…
Tencent Youtu Lab（腾讯优图实验室）；Sun Yat-sen University（中山大学）；Southern University of Science and Technology（南方科技大学）
Recently, the research interest of person re-identification (ReID) has gradually turned to video-based methods, which acquire a person representation by aggregating frame features of an entire video. However, existing video-based ReID methods do not consider the semantic difference brought by the outputs of different network stages, which potentially compromises the information richness of the person features. Furthermore, traditional methods ignore important relationship among frames, which causes information redundancy in fusion along the time axis. To address these issues, we propose a nove…
Massachusetts Institute of Technology（麻省理工学院）；University of California, San Diego（加州大学圣地亚哥分校）
The Poisson-sampling technique eliminates dependencies among symbol appearances in a random sequence. It has been used to simplify the analysis and strengthen the performance guarantees of randomized algorithms. Applying this method to universal compression, we relate the redundancies of fixed-length and Poisson-sampled sequences, use the relation to derive a simple single-letter formula that approximates the redundancy of any envelope class to within an additive logarithmic term. As a first application, we consider i.i.d. distributions over a small alphabet as a step-envelope class, and provi…
Smart Scissor: Coupling Spatial Redundancy Reduction and CNN Compression for Embedded Hardware
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；HP-NTU Digital Manufacturing Corporate Lab, Nanyang Technological University（惠普-南洋理工大学数字制造联合实验室）；HP Inc.（惠普公司）
Rethinking the Potential of Layer Freezing for Efficient DNN Training
University of Georgia（佐治亚大学）；Northeastern University（东北大学）；University of Pittsburgh（匹兹堡大学）；The University of Arizona（亚利桑那大学）；Stevens Institute of Technology（斯蒂文斯理工学院）；The University of North Carolina at Charlotte（北卡罗来纳大学教堂山分校）；University of Virginia（弗吉尼亚大学）
SenseTime Research（商汤科技研究院）；The University of Hong Kong（香港大学）；Sun Yat-Sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）
Channel pruning is a popular technique for compressing convolutional neural networks (CNNs), where various pruning criteria have been proposed to remove the redundant filters. From our comprehensive experiments, we found two blind spots in the study of pruning criteria: (1) Similarity: There are some strong similarities among several primary pruning criteria that are widely cited and compared. According to these criteria, the ranks of filters'Importance Score are almost identical, resulting in similar pruned structures. (2) Applicability: The filters'Importance Score measured by some pruning c…
Rethinking Dimensional Rationale in Graph Contrastive Learning from Causal Perspective
Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences（中国科学院软件研究所综合信息系统技术国家级重点实验室）；State Key Laboratory of Intelligent Game（智能博弈国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；State Key Laboratory of Computer Science, Institute of Software Chinese Academy of Sciences（中国科学院软件研究所计算机科学国家重点实验室）
The University of Tennessee Knoxville（田纳西大学诺克斯维尔分校）；Sun Yat-sen University（中山大学）
Convolutional neural network (CNN) pruning has become one of the most successful network compression approaches in recent years. Existing works on network pruning usually focus on removing the least important filters in the network to achieve compact architectures. In this study, we claim that identifying structural redundancy plays a more essential role than finding unimportant filters, theoretically and empirically. We first statistically model the network pruning problem in a redundancy reduction perspective and find that pruning in the layer(s) with the most structural redundancy outperfor…
Trinity College Dublin（都柏林圣三一学院）；Dublin City University（都柏林城市大学）
We show how parameter redundancy in Convolutional Neural Network (CNN) filters can be effectively reduced by pruning in spectral domain. Specifically, the representation extracted via Discrete Cosine Transform (DCT) is more conducive for pruning than the original space. By relying on a combination of weight tensor reshaping and reordering we achieve high levels of layer compression with just minor accuracy loss. Our approach is applied to compress pretrained CNNs and we show that minor additional fine-tuning allows our method to recover the original model performance after a significant parame…
PACC: Protocol-Aware Cross-Layer Compression for Compact Network Traffic Representation
University of Electronic Science and Technology of China（电子科技大学）；University of Hong Kong（香港大学）；Renmin University of China（中国人民大学）；Hong Kong Polytechnic University（香港理工大学）
