Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML
European Organization for Nuclear Research (CERN)（欧洲核子研究组织（CERN））；Institute of Physics Belgrade（贝尔格莱德物理研究所）；Columbia University（哥伦比亚大学）；University of California San Diego（加州大学圣地亚哥分校）；Massachusetts Institute of Technology（麻省理工学院）；Fermi National Accelerator Laboratory（费米国家加速器实验室）；HawkEye360；Boston University（波士顿大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Rhodes College（罗德学院）
Giuseppe Di Guglielmo, Javier Duarte, Philip Harris, Duc Hoang, Sergo Jindariani, Edward Kreinar, Mia Liu, Vladimir Loncar, Jennifer Ngadiuba, Kevin Pedro, Maur… 展开作者
Giuseppe Di Guglielmo, Javier Duarte, Philip Harris, Duc Hoang, Sergo Jindariani, Edward Kreinar, Mia Liu, Vladimir Loncar, Jennifer Ngadiuba, Kevin Pedro, Maurizio Pierini, Dylan Rankin, Sheila Sagear, Sioni Summers, Nhan Tran, Zhenbin Wu
We present the implementation of binary and ternary neural networks in the hls4ml library, designed to automatically convert deep neural network models to digital circuits with FPGA firmware. Starting from benchmark models trained with floating point precision, we investigate different strategies to reduce the network's resource consumption by reducing the numerical precision of the network parameters to binary or ternary. We discuss the trade-off between model accuracy and resource consumption. In addition, we show how to balance between latency and accuracy by retaining full precision on a selected subset of network components. As an example, we consider two multiclass classification tasks: handwritten digit recognition with the MNIST data set and jet identification with simulated proton-proton collisions at the CERN Large Hadron Collider. The binary and ternary implementation has similar performance to the higher precision implementation while using drastically fewer FPGA resources.
We demonstrate that 1x1-convolutions in 1D time-channel separable convolutions may be replaced by constant, sparse random ternary matrices with weights in $\{-1,0,+1\}$. Such layers do not perform any multiplications and do not require training. Moreover, the matrices may be generated on the chip during computation and therefore do not require any memory access. With the same parameter budget, we can afford deeper and more expressive models, improving the Pareto frontiers of existing models on several tasks. For command recognition on Google Speech Commands v1, we improve the state-of-the-art …
The University Of Sydney（悉尼大学）；Fulda University Of Applied Sciences（富尔达应用科学大学）；University Of Kassel（卡塞尔大学）
The computational complexity of neural networks for large scale or real-time applications necessitates hardware acceleration. Most approaches assume that the network architecture and parameters are unknown at design time, permitting usage in a large number of applications. This paper demonstrates, for the case where the neural network architecture and ternary weight values are known a priori, that extremely high throughput implementations of neural network inference can be made by customising the datapath and routing to remove unnecessary computations and data movement. This approach is ideall…
Compressing Recurrent Neural Networks for FPGA-accelerated Implementation in Fluorescence Lifetime Imaging
Center for Modeling, Simulation, & Imaging in Medicine（医学建模、仿真与成像中心）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM T.J. Watson Research Center（IBM TJ·沃森研究中心）
University of Electronic Science and Technology of China（电子科技大学）；Harvard University（哈佛大学）；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）
To deploy deep neural networks on resource-limited devices, quantization has been widely explored. In this work, we study the extremely low-bit networks which have tremendous speed-up, memory saving with quantized activation and weights. We first bring up three omitted issues in extremely low-bit networks: the squashing range of quantized values; the gradient vanishing during backpropagation and the unexploited hardware acceleration of ternary networks. By reparameterizing quantized activation and weights vector with full precision scale and offset for fixed ternary vector, we decouple the ran…
WeBank（微众银行）；Harbin Engineering University（哈尔滨工程大学）；University of Hyogo（兵库县立大学）；Gakushuin University（学习院大学）
Quantization of weights of deep neural networks (DNN) has proven to be an effective solution for the purpose of implementing DNNs on edge devices such as mobiles, ASICs and FPGAs, because they have no sufficient resources to support computation involving millions of high precision weights and multiply-accumulate operations. This paper proposes a novel method to compress vectors of high precision weights of DNNs to ternary vectors, namely a cosine similarity based target non-retraining ternary (TNT) compression method. Our method leverages cosine similarity instead of Euclidean distances as com…
MobileNets family of computer vision neural networks have fueled tremendous progress in the design and organization of resource-efficient architectures in recent years. New applications with stringent real-time requirements on highly constrained devices require further compression of MobileNets-like already compute-efficient networks. Model quantization is a widely used technique to compress and accelerate neural network inference and prior works have quantized MobileNets to 4-6 bits albeit with a modest to significant drop in accuracy. While quantization to sub-byte values (i.e. precision les…
CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
Syracuse University（雪城大学）；City University of New York, City College（纽约市立大学城市学院）；University of Southern California（南加州大学）；California State University Fullerton（加州州立大学富勒顿分校）；Northeastern University（东北大学）
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Federal Research Center “Computer Science and Control” of Russian Academy of Sciences（俄罗斯科学院计算机科学与控制联邦研究中心）；Smart Engines Service LLC（Smart Engines服务有限责任公司）；Institute for Information Transmission Problems, RAS（俄罗斯科学院信息传输问题研究所）
Low-bit quantized neural networks are of great interest in practical applications because they significantly reduce the consumption of both memory and computational resources. Binary neural networks are memory and computationally efficient as they require only one bit per weight and activation and can be computed using Boolean logic and bit count operations. QNNs with ternary weights and activations and binary weights and ternary activations aim to improve recognition quality compared to BNNs while preserving low bit-width. However, their efficient implementation is usually considered on ASICs…
Tsinghua University（清华大学）；Beijing University of Technology（北京工业大学）；Case Western Reserve University（凯斯西储大学）；BNRist（（无常见中文名，保留英文））；Dept. Computer Science & Tech., Tsinghua University（清华大学计算机科学与技术系）；Beijing Engineering Research Center for IoT Software and Systems, Beijing University of Technology（北京工业大学物联网软件与系统北京市工程研究中心）；Department of Electrical Engineering and Computer Science, Case Western Reserve University（凯斯西储大学电气工程与计算机科学系）
In this work, we propose an effective scheme (called DP-Net) for compressing the deep neural networks (DNNs). It includes a novel dynamic programming (DP) based algorithm to obtain the optimal solution of weight quantization and an optimization process to train a clustering-friendly DNN. Experiments showed that the DP-Net allows larger compression than the state-of-the-art counterparts while preserving accuracy. The largest 77X compression ratio on Wide ResNet is achieved by combining DP-Net with other compression techniques. Furthermore, the DP-Net is extended for compressing a robust DNN mod…
Convolutional neural networks are modern models that are very efficient in many classification tasks. They were originally created for image processing purposes. Then some trials were performed to use them in different domains like natural language processing. The artificial intelligence systems (like humanoid robots) are very often based on embedded systems with constraints on memory, power consumption etc. Therefore convolutional neural network because of its memory capacity should be reduced to be mapped to given hardware. In this paper, results are presented of compressing the efficient co…
Threshold Neuron: A Brain-inspired Artificial Neuron for Efficient On-device Inference
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；School of Computer Science, Peking University（北京大学计算机学院）；LuxiTech (2024)（LuxiTech（2024））
In this paper the research on optimisation of visual object tracking using a Siamese neural network for embedded vision systems is presented. It was assumed that the solution shall operate in real-time, preferably for a high resolution video stream, with the lowest possible energy consumption. To meet these requirements, techniques such as the reduction of computational precision and pruning were considered. Brevitas, a tool dedicated for optimisation and quantisation of neural networks for FPGA implementation, was used. A number of training scenarios were tested with varying levels of optimis…
