We propose a joint feature compression and transmission scheme for efficient inference at the wireless network edge. Our goal is to enable efficient and reliable inference at the edge server assuming limited computational resources at the edge device. Previous work focused mainly on feature compression, ignoring the computational cost of channel coding. We incorporate the recently proposed deep joint source-channel coding (DeepJSCC) scheme, and combine it with novel filter pruning strategies aimed at reducing the redundant complexity from neural networks. We evaluate our approach on a classification task, and show improved results in both end-to-end reliability and workload reduction at the edge device. This is the first work that combines DeepJSCC with network pruning, and applies it to image classification over the wireless edge.
Hong Kong University of Science and Technology（香港科技大学）；MEGVII Technology（旷视科技）；Carnegie Mellon University（卡内基梅隆大学）
We present joint multi-dimension pruning (abbreviated as JointPruning), an effective method of pruning a network on three crucial aspects: spatial, depth and channel simultaneously. To tackle these three naturally different dimensions, we proposed a general framework by defining pruning as seeking the best pruning vector (i.e., the numerical value of layer-wise channel number, spacial size, depth) and construct a unique mapping from the pruning vector to the pruned network structures. Then we optimize the pruning vector with gradient update and model joint pruning as a numerical gradient optim…
Quantization and pruning are core techniques used to reduce the inference costs of deep neural networks. State-of-the-art quantization techniques are currently applied to both the weights and activations; however, pruning is most often applied to only the weights of the network. In this work, we jointly apply novel uniform quantization and unstructured pruning methods to both the weights and activations of deep neural networks during training. Using our methods, we empirically evaluate the currently accepted prune-then-quantize paradigm across a wide range of computer vision tasks and observe …
Qualcomm AI Research（高通人工智能研究院）；University of California, Irvine（加利福尼亚大学尔湾分校）
We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between model pruning and quantization automatically for hardware efficiency. DJPQ incorporates variational information bottleneck based structured pruning and mixed-bit precision quantization into a single differentiable loss function. In contrast to previous works which consider pruning and quantization separately, our method enables users to find the optimal trade-off between both in a single training procedure. To utilize …
When approaching a novel visual recognition problem in a specialized image domain, a common strategy is to start with a pre-trained deep neural network and fine-tune it to the specialized domain. If the target domain covers a smaller visual space than the source domain used for pre-training (e.g. ImageNet), the fine-tuned network is likely to be over-parameterized. However, applying network pruning as a post-processing step to reduce the memory requirements has drawbacks: fine-tuning and pruning are performed independently; pruning parameters are set once and cannot adapt over time; and the hi…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械与车辆学院）；University of Chinese Academy of Sciences（中国科学院大学）；College of Engineering and Computer Science, Australian National University（澳大利亚国立大学工程与计算机科学学院）
Currently, an increasing number of model pruning methods are proposed to resolve the contradictions between the computer powers required by the deep learning models and the resource-constrained devices. However, most of the traditional rule-based network pruning methods can not reach a sufficient compression ratio with low accuracy loss and are time-consuming as well as laborious. In this paper, we propose Automatic Block-wise and Channel-wise Network Pruning (ABCP) to jointly search the block-wise and channel-wise pruning action with deep reinforcement learning. A joint sample algorithm is pr…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kwai Inc（快手公司）
For practical deep neural network design on mobile devices, it is essential to consider the constraints incurred by the computational resources and the inference latency in various applications. Among deep network acceleration related approaches, pruning is a widely adopted practice to balance the computational resource consumption and the accuracy, where unimportant connections can be removed either channel-wisely or randomly with a minimal impact on model accuracy. The channel pruning instantly results in a significant latency reduction, while the random weight pruning is more flexible to ba…
We propose joint transmission-recognition schemes for efficient inference at the wireless edge. Motivated by the surveillance applications with wireless cameras, we consider the person classification task over a wireless channel carried out by multi-view cameras operating as edge devices. We introduce deep neural network (DNN) based compression schemes which incorporate digital (separate) transmission and joint source-channel coding (JSCC) methods. We evaluate the proposed device-edge communication schemes under different channel SNRs, bandwidth and power constraints. We show that the JSCC sch…
Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
We propose Joint-DetNAS, a unified NAS framework for object detection, which integrates 3 key components: Neural Architecture Search, pruning, and Knowledge Distillation. Instead of naively pipelining these techniques, our Joint-DetNAS optimizes them jointly. The algorithm consists of two core processes: student morphism optimizes the student's architecture and removes the redundant parameters, while dynamic distillation aims to find the optimal matching teacher. For student morphism, weight inheritance strategy is adopted, allowing the student to flexibly update its architecture while fully u…
Duke University（杜克大学）；Black Sesame Technologies（黑芝麻科技）
With the rapid scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for improving deployment efficiency. This work aims to advance the compression beyond the weights to neuron activations. We propose the joint regularization technique which simultaneously regulates the distribution of weights and activations. By distinguishing and leveraging the significance difference among neuron responses and connections during learning, the jointly pruned network, namely \textit{JPnet}, optimizes the sparsity of activa…
We propose a simple but effective data-driven channel pruning algorithm, which compresses deep neural networks in a differentiable way by exploiting the characteristics of operations. The proposed approach makes a joint consideration of batch normalization (BN) and rectified linear unit (ReLU) for channel pruning; it estimates how likely the two successive operations deactivate each feature map and prunes the channels with high probabilities. To this end, we learn differentiable masks for individual channels and make soft decisions throughout the optimization procedure, which facilitates to ex…
Massachusetts Institute of Technology（麻省理工学院）；Shanghai Jiao Tong University（上海交通大学）
We present APQ for efficient deep learning inference on resource-constrained hardware. Unlike previous methods that separately search the neural architecture, pruning policy, and quantization policy, we optimize them in a joint manner. To deal with the larger design space it brings, a promising approach is to train a quantization-aware accuracy predictor to quickly get the accuracy of the quantized model and feed it to the search engine to select the best fit. However, training this quantization-aware accuracy predictor requires collecting a large number of quantized <model, accuracy> pairs, w…
Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks
Institute for Infocomm Research (IR), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR）信息通信研究院（IR））；Nanyang Technological University, Singapore（新加坡南洋理工大学）
Deep convolutional neural networks are shown to be overkill with high parametric and computational redundancy in many application scenarios, and an increasing number of works have explored model pruning to obtain lightweight and efficient networks. However, most existing pruning approaches are driven by empirical heuristic and rarely consider the joint impact of channels, leading to unguaranteed and suboptimal performance. In this paper, we propose a novel channel pruning method via Class-Aware Trace Ratio Optimization (CATRO) to reduce the computational burden and accelerate the model inferen…
Unstructured neural network pruning algorithms have achieved impressive compression rates. However, the resulting - typically irregular - sparse matrices hamper efficient hardware implementations, leading to additional memory usage and complex control logic that diminishes the benefits of unstructured pruning. This has spurred structured coarse-grained pruning solutions that prune entire filters or even layers, enabling efficient implementation at the expense of reduced flexibility. Here we propose a flexible new pruning mechanism that facilitates pruning at different granularities (weights, k…
Robert Bosch GmbH（罗伯特·博世有限公司）；University of Lübeck（吕贝克大学）
Unstructured pruning is well suited to reduce the memory footprint of convolutional neural networks (CNNs), both at training and inference time. CNNs contain parameters arranged in $K \times K$ filters. Standard unstructured pruning (SP) reduces the memory footprint of CNNs by setting filter elements to zero, thereby specifying a fixed subspace that constrains the filter. Especially if pruning is applied before or during training, this induces a strong bias. To overcome this, we introduce interspace pruning (IP), a general tool to improve existing pruning methods. It uses filters represented i…
SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration
Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Beihang University（北京航空航天大学）
Xiamen University（厦门大学）；Beihang University（北京航空航天大学）；University of Chinese Academy of Sciences（中国科学院大学）；BestImage, Tencent Technology (Shanghai) Co.,Ltd（腾讯科技（上海）有限公司贝思图）
Structured pruning of filters or neurons has received increased focus for compressing convolutional neural networks. Most existing methods rely on multi-stage optimizations in a layer-wise manner for iteratively pruning and retraining which may not be optimal and may be computation intensive. Besides, these methods are designed for pruning a specific structure, such as filter or block structures without jointly pruning heterogeneous structures. In this paper, we propose an effective structured pruning approach that jointly prunes filters as well as other structures in an end-to-end manner. To …
While convolutional neural networks (CNN) have achieved impressive performance on various classification/recognition tasks, they typically consist of a massive number of parameters. This results in significant memory requirement as well as computational overheads. Consequently, there is a growing need for filter-level pruning approaches for compressing CNN based models that not only reduce the total number of parameters but reduce the overall computation as well. We present a new min-max framework for filter-level pruning of CNNs. Our framework, called Play and Prune (PP), jointly prunes and f…
In this paper, we analyze two popular network compression techniques, i.e. filter pruning and low-rank decomposition, in a unified sense. By simply changing the way the sparsity regularization is enforced, filter pruning and low-rank decomposition can be derived accordingly. This provides another flexible choice for network compression because the techniques complement each other. For example, in popular network architectures with shortcut connections (e.g. ResNet), filter pruning cannot deal with the last convolutional layer in a ResBlock while the low-rank decomposition methods can. In addit…
ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Models
VCIP & TMCC & DISSec, College of Computer Science, Nankai University（VCIP与TMCC与DISSec学院，南开大学计算机科学学院）；LIX, Ecole Polytechnique, IP Paris（LIX，巴黎高等理工学院，IP巴黎）；OPPO Research Institute（OPPO研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
National Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；Nanjing University（南京大学）；Nanjing（南京）；China（中国）
Channel pruning is an important family of methods to speed up deep model's inference. Previous filter pruning algorithms regard channel pruning and model fine-tuning as two independent steps. This paper argues that combining them into a single end-to-end trainable system will lead to better results. We propose an efficient channel selection layer, namely AutoPruner, to find less important filters automatically in a joint training manner. Our AutoPruner takes previous activation responses as an input and generates a true binary index code for pruning. Hence, all the filters corresponding to zer…
George Washington University（乔治华盛顿大学）；Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）
One of the biggest challenges in Federated Learning (FL) is that client devices often have drastically different computation and communication resources for local updates. To this end, recent research efforts have focused on training heterogeneous local models obtained by pruning a shared global model. Despite empirical success, theoretical guarantees on convergence remain an open question. In this paper, we present a unifying framework for heterogeneous FL algorithms with {\em arbitrary} adaptive online model pruning and provide a general convergence analysis. In particular, we prove that und…
Introducing sparsity in a neural network has been an efficient way to reduce its complexity while keeping its performance almost intact. Most of the time, sparsity is introduced using a three-stage pipeline: 1) train the model to convergence, 2) prune the model according to some criterion, 3) fine-tune the pruned model to recover performance. The last two steps are often performed iteratively, leading to reasonable results but also to a time-consuming and complex process. In our work, we propose to get rid of the first step of the pipeline and to combine the two other steps in a single pruning…
University of York（约克大学）；PUC-Rio（里约热内卢天主教大学）；Princeton University（普林斯顿大学）
This paper proposes a novel distributed reduced--rank scheme and an adaptive algorithm for distributed estimation in wireless sensor networks. The proposed distributed scheme is based on a transformation that performs dimensionality reduction at each agent of the network followed by a reduced-dimension parameter vector. A distributed reduced-rank joint iterative estimation algorithm is developed, which has the ability to achieve significantly reduced communication overhead and improved performance when compared with existing techniques. Simulation results illustrate the advantages of the propo…
Motivated by surveillance applications with wireless cameras or drones, we consider the problem of image retrieval over a wireless channel. Conventional systems apply lossy compression on query images to reduce the data that must be transmitted over the bandwidth and power limited wireless link. We first note that reconstructing the original image is not needed for retrieval tasks; hence, we introduce a deep neutral network (DNN) based compression scheme targeting the retrieval task. Then, we completely remove the compression step, and propose another DNN-based communication scheme that direct…
The demand for flexible broadband wireless services makes the pruning technique, including both shortening and puncturing, an indispensable component of error correcting codes. The analysis of the pruning process for structured lowdensity parity-check (LDPC) codes can be considerably simplified with their equivalent representations through base-matrices or protographs. In this letter, we evaluate the thresholds of the pruned base-matrices by using protograph based on extrinsic information transfer (PEXIT). We also provide an efficient method to optimize the pruning patterns, which can signific…
In this paper, we provide an overview of the recent advances in microwave-enabled wireless energy transfer (WET) technologies and their applications in wireless communications. Specifically, we divide our discussions into three parts. First, we introduce the state-of-the-art WET technologies and the signal processing techniques to maximize the energy transfer efficiency. Then, we discuss an interesting paradigm named simultaneous wireless information and power transfer (SWIPT), where energy and information are jointly transmitted using the same radio waveform. At last, we review the recent pro…
In this paper, a novel approach for optimizing the joint deployment of small cell base stations and wireless backhaul links is proposed. This joint deployment scenario is cast as a multi-objective optimization problem under the constraints of limited backhaul capacity and outage probability. To address the problem,a novel adaptive algorithm that integrates $ε$-method, Lagrangian relaxation and tabu search is proposed to obtain the Pareto optimal solution set. Simulation results show that the proposed algorithm is quite effective in finding the optimal solutions. The proposed joint deployment m…
Resource management, including power control, is one of the most essential functionalities of any wireless telecommunication system. Various transmitter power-control methods have been developed to deliver a desired quality of service in wireless networks. We consider two of these methods: Distributed Power Control and Distributed Balancing Algorithm schemes. We use the concept of joint spectral radius to come up with conditions for convergence of the transmitted power in these two schemes when the gains on all the communications links are assumed to vary at each time-step.
In this paper, we investigate the joint impact of misalignment fading and local oscillator (LO) phase noise (PHN) in multi-carrier terahertz (THz) wireless systems. In more detail, after establishing a suitable system model that takes into account the particularities of the THz channel, as well as the transceivers characteristics, we present simulation results that quantify the joint impact of misalignment fading and PHN in terms of average signal-to-interference-plus-noise-ratio (SINR) and outage probability (OP).
Beijing University of Posts and Telecommunications（北京邮电大学）
In this letter, we study a wireless federated learning (FL) system where network pruning is applied to local users with limited resources. Although pruning is beneficial to reduce FL latency, it also deteriorates learning performance due to the information loss. Thus, a trade-off problem between communication and learning is raised. To address this challenge, we quantify the effects of network pruning and packet error on the learning performance by deriving the convergence rate of FL with a non-convex loss function. Then, closed-form solutions for pruning control and bandwidth allocation are p…
We demonstrate a high capacity IF-over-fiber mobile fronthaul solution for multi-user massive MIMO 5G RAN. Using this scheme, a record aggregated radio bandwidth of 25.6 GHz was transmitted on a single optical wavelength over 40 km without fiber chromatic dispersion compensation
The fifth generation (5G) communication scenarios such as the cellular network and the emerging machine type communications will produce massive small packets. To support massive connectivity and avoid signaling overhead caused by the transmission of those small packets, this paper proposes a novel method to improve the transmission efficiency for massive connections of wireless uplink channel. The proposed method combines compressive sensing (CS) with power domain NOMA jointly, especially neither the scheduling nor the centralized power allocation is necessary in the method. Both the analysis…
Far-field Wireless Power Transfer (WPT) has emerged as a potential power source for the Internet of Things (IoT) and Wireless Sensor Network (WSN).The expansion of the power transfer range is one of the key challenges to make the technology viable. In this paper, we experimentally study a channel-adaptive joint beamforming and waveform architecture to expand the power transfer range. WPT experiments have been conducted in a variety of wireless channels at various distances in a realistic indoor environment. The measurement data have been fitted using a simple analytical model to analyze the ou…
In this paper, we consider a multihop wireless sensor network (WSN) with multiple relay nodes for each hop where the amplify-and-forward (AF) scheme is employed. We present a strategy to jointly design the linear receiver and the power allocation parameters via an alternating optimization approach that maximizes the sum rate of the WSN. We derive constrained maximum sum-rate (MSR) expressions along with an algorithm to compute the linear receiver and the power allocation parameters with the optimal complex amplification coefficients for each relay node. Computer simulations show good performan…
In this paper, we investigate the joint design of channel and network coding in bi-directional relaying systems and propose a combined low complexity physical network coding and LDPC decoding scheme. For the same LDPC codes employed at both source nodes, we show that the relay can decodes the network coded codewords from the superimposed signal received from the BPSK-modulated multiple-access channel. Simulation results shown that this novel joint physical network coding and LDPC decoding method outperforms the existing MMSE network coding and LDPC decoding method over AWGN and complex MAC cha…
In this report we present an analysis of the non-random and the Bayesian Cramer-Rao lower bound (CRLB) for the joint estimation of angle-of-arrival (AoA), angle-of-departure (AoD), and the multipath amplitudes, for the millimeter-wave (mmWave) wireless networks. Our analysis is applicable to multipath channels with Gaussian noise and independent path parameters. Numerical results based on uniform AoA and AoD in $[0,π)$, and Rician fading path amplitudes, reveal that the Bayesian CRLB decreases monotonically with an increase in the Rice factor. Further, the CRLB obtained by using beamforming an…
In this work, we maximize the secrecy rate of the wireless-powered untrusted relay network by jointly designing power splitting (PS) ratio and relay beamforming with the proposed global optimal algorithm (GOA) and local optimal algorithm (LOA). Different from the literature, artificial noise (AN) sent by the destination not only degrades the channel condition of the eavesdropper to improve the secrecy rate, but also becomes a new source of energy powering the untrusted relay based on PS. Hence, it is of high economic benefits and efficiency to take advantage of AN compared with the literature.…
