In this work we present a new framework for neural networks compression with fine-tuning, which we called Neural Network Compression Framework (NNCF). It leverages recent advances of various network compression methods and implements some of them, such as sparsity, quantization, and binarization. These methods allow getting more hardware-friendly models which can be efficiently run on general-purpose hardware computation units (CPU, GPU) or special Deep Learning accelerators. We show that the developed methods can be successfully applied to a wide range of models to accelerate the inference time while keeping the original accuracy. The framework can be used within the training samples, which are supplied with it, or as a standalone package that can be seamlessly integrated into the existing training code with minimal adaptations. Currently, a PyTorch version of NNCF is available as a part of OpenVINO Training Extensions at https://github.com/openvinotoolkit/nncf.
Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）
Video-to-Video synthesis (Vid2Vid) has achieved remarkable results in generating a photo-realistic video from a sequence of semantic maps. However, this pipeline suffers from high computational cost and long inference latency, which largely depends on two essential factors: 1) network architecture parameters, 2) sequential data stream. Recently, the parameters of image-based generative models have been significantly compressed via more efficient network architectures. Nevertheless, existing methods mainly focus on slimming network architectures and ignore the size of the sequential data stream…
It has been witnessed that learned image compression has outperformed conventional image coding techniques and tends to be practical in industrial applications. One of the most critical issues that need to be considered is the non-deterministic calculation, which makes the probability prediction cross-platform inconsistent and frustrates successful decoding. We propose to solve this problem by introducing well-developed post-training quantization and making the model inference integer-arithmetic-only, which is much simpler than presently existing training and fine-tuning based approaches yet s…
Wavefront Parallelization for Efficient Learned Image Compression
School of Fundamental Science（基础科学学校）；Engineering Waseda University Tokyo, Japan（东京日本早稻田大学工程）；School of Engineering Institute of Science Tokyo Tokyo, Japan（东京东京科学研究所工程学校）
ETH Zürich（苏黎世联邦理工学院）；University of Bologna（博洛尼亚大学）
In the wake of the success of convolutional neural networks in image classification, object recognition, speech recognition, etc., the demand for deploying these compute-intensive ML models on embedded and mobile systems with tight power and energy constraints at low cost, as well as for boosting throughput in data centers, is growing rapidly. This has sparked a surge of research into specialized hardware accelerators. Their performance is typically limited by I/O bandwidth, power consumption is dominated by I/O transfers to off-chip memory, and on-chip memories occupy a large part of the sili…
Most action recognition solutions rely on dense sampling to precisely cover the informative temporal clip. Extensively searching temporal region is expensive for a real-world application. In this work, we focus on improving the inference efficiency of current action recognition backbones on trimmed videos, and illustrate that one action model can also cover then informative region by dropping non-informative features. We present Selective Feature Compression (SFC), an action recognition inference strategy that greatly increase model inference efficiency without any accuracy compromise. Differe…
This paper studies the computational offloading of CNN inference in device-edge co-inference systems. Inspired by the emerging paradigm semantic communication, we propose a novel autoencoder-based CNN architecture (AECNN), for effective feature extraction at end-device. We design a feature compression module based on the channel attention method in CNN, to compress the intermediate data by selecting the most important features. To further reduce communication overhead, we can use entropy encoding to remove the statistical redundancy in the compressed data. At the receiver, we design a lightwei…
Convolution neural network demonstrates great capability for multiple tasks, such as image classification and many others. However, much resource is required to train a network. Hence much effort has been made to accelerate neural network by reducing precision of weights, activation, and gradient. However, these filter-wise quantification methods exist a natural upper limit, caused by the size of the kernel. Meanwhile, with the popularity of small kernel, the natural limit further decrease. To address this issue, we propose a new cross-filter compression method that can provide $\sim32\times$ …
Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inference
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；China Telecom（中国电信）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
University of California Los Angeles（加利福尼亚大学洛杉矶分校）；Apple Inc.（苹果公司）
When deploying deep learning models to a device, it is traditionally assumed that available computational resources (compute, memory, and power) remain static. However, real-world computing systems do not always provide stable resource guarantees. Computational resources need to be conserved when load from other processes is high or battery power is low. Inspired by recent works on neural network subspaces, we propose a method for training a "compressible subspace" of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. Our models…
Transform image processing methods are methods that work in domains of image transforms, such as Discrete Fourier, Discrete Cosine, Wavelet and alike. They are the basic tool in image compression, in image restoration, in image re-sampling and geometrical transformations and can be traced back to early 1970-ths. The paper presents a review of these methods with emphasis on their comparison and relationships, from the very first steps of transform image compression methods to adaptive and local adaptive transform domain filters for image restoration, to methods of precise image re-sampling and …
NeuCODEX: Edge-Cloud Co-Inference with Spike-Driven Compression and Dynamic Early-Exit
Centre for Sustainable Digital Technologies（可持续数字技术中心）；Technological University Dublin（技术大学都柏林）；Smart Network Innovation Lab（智能网络创新实验室）；Huawei Ireland Research Center（华为爱尔兰研究中心）
The primary goal of this work is to review the importance of data compression and present a fast Fourier-based method for generating the deterministic compression matrix in the area of deterministic compressed sensing. The principle concepts of data compression such as general process of data compression, sparse signals, coherence matrix and Restricted Isometry Property (RIP) have been defined. We have introduced two methods of sparse data compression. The first method is formed by utilizing a stochastic matrix which is a common approach, and the second method is created by utilizing a determi…
Universidad Nacional de Tres de Febrero（特雷斯德费布雷罗国立大学）
In this work, we present a comparison between two techniques of image compression. In the first case, the image is divided in blocks which are collected according to zig-zag scan. In the second one, we apply the Fast Cosine Transform to the image, and then the transformed image is divided in blocks which are collected according to zig-zag scan too. Later, in both cases, the Karhunen-Loeve transform is applied to mentioned blocks. On the other hand, we present three new metrics based on eigenvalues for a better comparative evaluation of the techniques. Simulations show that the combined version…
Intel – Artificial Intelligence Products Group (AIPG)（英特尔人工智能产品事业部）；Technion – Israel Institute of Technology（以色列理工学院）
Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving visual processing tasks. One of the major obstacles hindering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be a main energy consumer and throughput bottleneck in hardware accelerators. Accordingly, an efficient feature map compression method can result in substantial performance gains. Inspired by quantization-aware training approaches, we propose a compression-aware training (CAT) method that involves training the model in a way th…
In many learning situations, resources at inference time are significantly more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient at inference time. Given a resource-intensive base architecture, DARC utilizes the training data to learn which sub-components can be replaced by cheaper alternatives. The high-level technique can be applied to any neural architecture, and we report experiments on state-of-the-art…
In this paper a new fractal image compression algorithm is proposed in which the time of encoding process is considerably reduced. The algorithm exploits a domain pool reduction approach, along with using innovative predefined values for contrast scaling factor, S, instead of scanning the parameter space [0,1]. Within this approach only domain blocks with entropies greater than a threshold are considered. As a novel point, it is assumed that in each step of the encoding process, the domain block with small enough distance shall be found only for the range blocks with low activity (equivalently…
Fractal image compression has some desirable properties like high quality at high compression ratio, fast decoding, and resolution independence. Therefore it can be used for many applications such as texture mapping and pattern recognition and image watermarking. But it suffers from long encoding time due to its need to find the best match between sub blocks. This time is related to the approach that is used. In this paper we present a fast encoding Algorithm based on no search method. Our goal is that more blocks are covered in initial step of quad tree algorithm. Experimental result has been…
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；Renmin University of China（中国人民大学）
By applying entropy codecs with learned data distributions, neural compressors have significantly outperformed traditional codecs in terms of compression ratio. However, the high inference latency of neural networks hinders the deployment of neural compressors in practical applications. In this work, we propose Integer-only Discrete Flows (IODF), an efficient neural compressor with integer-only arithmetic. Our work is built upon integer discrete flows, which consists of invertible transformations between discrete random variables. We propose efficient invertible transformations with integer-on…
Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference
Institute of Artificial Intelligence (TeleAI) of China Telecom（中国电信人工智能研究所）；School of Electronic and Information Engineering, Harbin Institute of Technology（哈尔滨工业大学电子与信息工程学院）；Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology（香港科技大学电子与计算机工程系）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）
On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
The Hong Kong Polytechnic University, Hong Kong（香港理工大学）；Nanyang Technological University, Singapore（南洋理工大学）；Chongqing University, Chongqing, China（重庆大学）；Research Centre for Privacy and Security Technologies in Future Smart Systems, PolyU（未来智能系统中的隐私与安全技术研究中心）
EdgeCompress: Coupling Multidimensional Model Compression and Dynamic Inference for EdgeAI
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；HP-NTU Digital Manufacturing Corporate Lab, Nanyang Technological University（惠普-南洋理工大学数字制造联合实验室）；Department of Computer Science, Norwegian University of Science and Technology（挪威科技大学计算机科学系）；HP Inc.（惠普公司）
Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a distributed inference paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communication-aware. Therefore, we propose Network of Neural Networks (NoNN), a new distributed IoT learning parad…
Georgia Institute of Technology（佐治亚理工学院）；Intel Corporation（英特尔公司）
The rapidly growing body of research in adversarial machine learning has demonstrated that deep neural networks (DNNs) are highly vulnerable to adversarially generated images. This underscores the urgent need for practical defense that can be readily deployed to combat attacks in real-time. Observing that many attack strategies aim to perturb image pixels in ways that are visually imperceptible, we place JPEG compression at the core of our proposed Shield defense framework, utilizing its capability to effectively "compress away" such pixel manipulation. To immunize a DNN model from artifacts i…
Xi’an Jiaotong University（西安交通大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Pittsburgh（匹兹堡大学）
Electron Cryo-Tomography (ECT) enables 3D visualization of macromolecule structure inside single cells. Macromolecule classification approaches based on convolutional neural networks (CNN) were developed to separate millions of macromolecules captured from ECT systematically. However, given the fast accumulation of ECT data, it will soon become necessary to use CNN models to efficiently and accurately separate substantially more macromolecules at the prediction stage, which requires additional computational costs. To speed up the prediction, we compress classification models into compact neura…
Ecole Centrale de Lyon（里昂中央理工学院）；Institut des Nanotechnologies de Lyon and CNRS（里昂纳米技术研究所及法国国家科学研究中心）
The computational workload involved in Convolutional Neural Networks (CNNs) is typically out of reach for low-power embedded devices. There are a large number of approximation techniques to address this problem. These methods have hyper-parameters that need to be optimized for each CNNs using design space exploration (DSE). The goal of this work is to demonstrate that the DSE phase time can easily explode for state of the art CNN. We thus propose the use of an optimized exploration process to drastically reduce the exploration time without sacrificing the quality of the output.
