arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.01344cs.LGcs.NE
收藏

二值化神经网络的比特容错性

Bit Error Tolerance Metrics for Binarized Neural Networks

Technical University of Dortmund(多特蒙德工业大学)

Sebastian Buschjäger, Jian-Jia Chen, Kuan-Hsun Chen, Mario Günzel, Katharina Morik, Rodion Novkin, Lukas Pfahler, Mikail Yayla

摘要:

为了减少神经网络(NN)推理系统的资源需求,提出了使用近似存储器,其中电源电压和时序参数被调整为交易精度与能耗和性能。积极地调整这些参数会导致比特错误,当在训练期间注入比特翻转时,NN可以容忍比特错误。然而,作为用于实现比特差错容限的现有技术的比特翻转训练不能很好地扩展;它导致大量开销并且不能应用于高比特出错率(BER)。需要替代方法来实现NN中的比特错误容限,但是NN的比特错误容限背后的基本原理尚未被报道。这种认识的缺乏,将制约神经网络比特容错性研究的进一步发展。在这项研究中,我们的目标是研究位翻转训练引起的NN的内部变化,重点是二值化NN(BNN)。为此,我们量化的比特错误容忍BNN的属性与两个指标。首先,我们提出了一个神经元级的位错误容忍度度量,它计算预激活值和批量归一化阈值之间的裕度。其次,为了捕捉比特差错容忍度对神经元之间相互作用的影响,我们提出了一种神经元间比特差错容忍度度量,该度量度量每个神经元的重要性并计算所有重要性值的方差。我们的实验结果支持,这两个指标是密切相关的比特错误容忍度。

英文摘要:

To reduce the resource demand of neural network (NN) inference systems, it has been proposed to use approximate memory, in which the supply voltage and the timing parameters are tuned trading accuracy with energy consumption and performance. Tuning these parameters aggressively leads to bit errors, which can be tolerated by NNs when bit flips are injected during training. However, bit flip training, which is the state of the art for achieving bit error tolerance, does not scale well; it leads to massive overheads and cannot be applied for high bit error rates (BERs). Alternative methods to achieve bit error tolerance in NNs are needed, but the underlying principles behind the bit error tolerance of NNs have not been reported yet. With this lack of understanding, further progress in the research on NN bit error tolerance will be restrained. In this study, our objective is to investigate the internal changes in the NNs that bit flip training causes, with a focus on binarized NNs (BNNs). To this end, we quantify the properties of bit error tolerant BNNs with two metrics. First, we propose a neuron-level bit error tolerance metric, which calculates the margin between the pre-activation values and batch normalization thresholds. Secondly, to capture the effects of bit error tolerance on the interplay of neurons, we propose an inter-neuron bit error tolerance metric, which measures the importance of each neuron and computes the variance over all importance values. Our experimental results support that these two metrics are strongly related to bit error tolerance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑