首页/论文/2004.03333

arXiv 2004.033332020-04cs.NEcs.CVcs.LG

Binary Neural Networks: A Survey

Beihang University（北京航空航天大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Trento（特伦托大学）

Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, Nicu Sebe

发表信息：: Pattern Recognition (2020) 107281
DOI：: 10.1016/j.patcog.2020.107281

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

The binary neural network, largely saving the storage and computation, serves as a promising technique for deploying deep models on resource-limited devices. However, the binarization inevitably causes severe information loss, and even worse, its discontinuity brings difficulty to the optimization of the deep network. To address these issues, a variety of algorithms have been proposed, and achieved satisfying progress in recent years. In this paper, we present a comprehensive survey of these algorithms, mainly categorized into the native solutions directly conducting binarization, and the optimized ones using techniques like minimizing the quantization error, improving the network loss function, and reducing the gradient error. We also investigate other practical aspects of binary neural networks such as the hardware-friendly design and the training tricks. Then, we give the evaluation and discussions on different tasks, including image classification, object detection and semantic segmentation. Finally, the challenges that may be faced in future research are prospected.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation

Singapore Management University（新加坡管理大学）；Columbia University（哥伦比亚大学）；Carnegie Mellon University（卡内基梅隆大学）

Compared to Multilayer Neural Networks with real weights, Binary Multilayer Neural Networks (BMNNs) can be implemented more efficiently on dedicated hardware. BMNNs have been demonstrated to be effective on binary classification tasks with Expectation BackPropagation (EBP) algorithm on high dimensional text datasets. In this paper, we investigate the capability of BMNNs using the EBP algorithm on multiclass image classification tasks. The performances of binary neural networks with multiple hidden layers and different numbers of hidden units are examined on MNIST. We also explore the effective…

2015-03-22cs.CVcs.LGcs.NE

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

École Polytechnique de Montréal（蒙特利尔综合理工学院）；Université de Montréal（蒙特利尔大学）

Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possib…

2016-04-18cs.CVcs.LGcs.NE

Controlling Information Capacity of Binary Neural Network

Russian Research Center Huawei Technologies, Russia（华为技术有限公司俄罗斯研究中心）；Department of Computer Science ETH Zurich, Switzerland（苏黎世联邦理工学院计算机科学系）

Despite the growing popularity of deep learning technologies, high memory requirements and power consumption are essentially limiting their application in mobile and IoT areas. While binary convolutional networks can alleviate these problems, the limited bitwidth of weights is often leading to significant degradation of prediction accuracy. In this paper, we present a method for training binary networks that maintains a stable predefined level of their information capacity throughout the training process by applying Shannon entropy based penalty to convolutional filters. The results of experim…

2020-08-04cs.CVcs.LGcs.NE

Probabilistic Binary-Mask Cocktail-Party Source Separation in a Convolutional Deep Neural Network

University of Surrey（萨里大学）

Separation of competing speech is a key challenge in signal processing and a feat routinely performed by the human auditory brain. A long standing benchmark of the spectrogram approach to source separation is known as the ideal binary mask. Here, we train a convolutional deep neural network, on a two-speaker cocktail party problem, to make probabilistic predictions about binary masks. Our results approach ideal binary mask performance, illustrating that relatively simple deep neural networks are capable of robust binary mask prediction. We also illustrate the trade-off between prediction stati…

2015-03-24cs.LGcs.NEcs.SD

TentacleNet: A Pseudo-Ensemble Template for Accurate Binary Convolutional Neural Networks

Politecnico di Torino（都灵理工大学）

Binarization is an attractive strategy for implementing lightweight Deep Convolutional Neural Networks (CNNs). Despite the unquestionable savings offered, memory footprint above all, it may induce an excessive accuracy loss that prevents a widespread use. This work elaborates on this aspect introducing TentacleNet, a new template designed to improve the predictive performance of binarized CNNs via parallelization. Inspired by the ensemble learning theory, it consists of a compact topology that is end-to-end trainable and organized to minimize memory utilization. Experimental results collected …

2019-12-26cs.CVcs.LGcs.NE

Transfer Learning with Binary Neural Networks

Ghent University - imec, IDLab, Department of Information Technology（根特大学 - imec，IDLab，信息技术系）；Ghent University（根特大学）

Previous work has shown that it is possible to train deep neural networks with low precision weights and activations. In the extreme case it is even possible to constrain the network to binary values. The costly floating point multiplications are then reduced to fast logical operations. High end smart phones such as Google's Pixel 2 and Apple's iPhone X are already equipped with specialised hardware for image processing and it is very likely that other future consumer hardware will also have dedicated accelerators for deep neural networks. Binary neural networks are attractive in this case bec…

2017-11-29cs.CVcs.NE

更多相关论文收起更多论文

BCNN: Binary Complex Neural Network

Zhejiang University（浙江大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）

Binarized neural networks, or BNNs, show great promise in edge-side applications with resource limited hardware, but raise the concerns of reduced accuracy. Motivated by the complex neural networks, in this paper we introduce complex representation into the BNNs and propose Binary complex neural network -- a novel network design that processes binary complex inputs and weights through complex convolution, but still can harvest the extraordinary computation efficiency of BNNs. To ensure fast convergence rate, we propose novel BCNN based batch normalization function and weight initialization fun…

2021-03-28cs.LGcs.NE

BiSNN: Training Spiking Neural Networks with Binary Weights via Bayesian Learning

King's College London（伦敦国王学院）

Artificial Neural Network (ANN)-based inference on battery-powered devices can be made more energy-efficient by restricting the synaptic weights to be binary, hence eliminating the need to perform multiplications. An alternative, emerging, approach relies on the use of Spiking Neural Networks (SNNs), biologically inspired, dynamic, event-driven models that enhance energy efficiency via the use of binary, sparse, activations. In this paper, an SNN model is introduced that combines the benefits of temporally sparse binary activations and of binary weights. Two learning rules are derived, the fir…

2020-12-15cs.LGcs.NEeess.SP

无需浮点精度训练二值神经网络

Training binary neural networks without floating point precision

Facoltà di Ingegneria dell’Informazione, Informatica e Statistica（工程、信息学与统计学学院）

本文针对二值神经网络训练效率问题，提出拓扑修改与策略训练两种方案，在接近最优性能的同时降低训练时间和内存需求。

2023-10-19cs.AIcs.LGcs.NE

Binary Weighted Memristive Analog Deep Neural Network for Near-Sensor Edge Processing

Nazarbayev University（纳扎尔巴耶夫大学）

The memristive crossbar aims to implement analog weighted neural network, however, the realistic implementation of such crossbar arrays is not possible due to limited switching states of memristive devices. In this work, we propose the design of an analog deep neural network with binary weight update through backpropagation algorithm using binary state memristive devices. We show that such networks can be successfully used for image processing task and has the advantage of lower power consumption and small on-chip area in comparison with digital counterparts. The proposed network was benchmark…

2018-08-02cs.AIcs.ETcs.NE

Dimensions of Neural-symbolic Integration - A Structured Survey

Research on integrated neural-symbolic systems has made significant progress in the recent past. In particular the understanding of ways to deal with symbolic knowledge within connectionist systems (also called artificial neural networks) has reached a critical mass which enables the community to strive for applicable implementations and use cases. Recent work has covered a great variety of logics used in artificial intelligence and provides a multitude of techniques for dealing with them within the context of artificial neural networks. We present a comprehensive survey of the field of neural…

2005-11-10cs.AIcs.LOcs.NE