使用集体低精度和结构化压缩最小化深度学习硬件设计的面积和能量
Minimizing Area and Energy of Deep Learning Hardware Design Using Collective Low Precision and Structured Compression
摘要:
深度学习算法在许多识别任务中取得了巨大的成功;然而,这些算法通常包括深度神经网络(DNN)结构和大量参数,这使得在功率/面积受限的嵌入式平台上实现它们具有挑战性。为了减少网络的大小,一些研究通过修剪和正则化引入元素或行/列/块稀疏性来研究压缩。此外,最近的许多工作都集中在降低激活和权重的精度上,其中一些降低到单个位。然而,将各种稀疏结构与二值化或非常低精度(2-3位)的神经网络相结合还没有被全面地探索。在这项工作中,我们提出了最小面积/能量DNN硬件的设计技术,其精度下降最小。在训练过程中,二值化/低精度和结构化稀疏性都被用作约束,以找到给定深度学习算法的最小内存占用。CIFAR-10数据集的DNN模型具有50倍的权重内存减少,其准确性与浮点对应模型相当。DNN硬件在40纳米CMOS的面积,性能和能量的结果报告的MNIST数据集。结合了8倍结构化压缩和3位权重精度的优化DNN在每个分类20 nJ时显示出98.4%的准确率。
英文摘要:
Deep learning algorithms have shown tremendous success in many recognition tasks; however, these algorithms typically include a deep neural network (DNN) structure and a large number of parameters, which makes it challenging to implement them on power/area-constrained embedded platforms. To reduce the network size, several studies investigated compression by introducing element-wise or row-/column-/block-wise sparsity via pruning and regularization. In addition, many recent works have focused on reducing precision of activations and weights with some reducing down to a single bit. However, combining various sparsity structures with binarized or very-low-precision (2-3 bit) neural networks have not been comprehensively explored. In this work, we present design techniques for minimum-area/-energy DNN hardware with minimal degradation in accuracy. During training, both binarization/low-precision and structured sparsity are applied as constraints to find the smallest memory footprint for a given deep learning algorithm. The DNN model for CIFAR-10 dataset with weight memory reduction of 50X exhibits accuracy comparable to that of the floating-point counterpart. Area, performance and energy results of DNN hardware in 40nm CMOS are reported for the MNIST dataset. The optimized DNN that combines 8X structured compression and 3-bit weight precision showed 98.4% accuracy at 20nJ per classification.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。