arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.06984cs.LG
收藏

来自ANN的免费午餐:实现高效、准确的脉冲神经网络校准

A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration

University of Electronic Science and Technology of China(电子科技大学);Yale University(耶鲁大学);Harvard University(哈佛大学);SenseTime Research(商汤科技研究院)

Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, Shi Gu

摘要:

脉冲神经网络(SNN)被认为是下一代神经网络的代表。传统上,SNN可以从预先训练的ANN转换,只需将ReLU激活替换为尖峰激活,同时保持参数不变。也许令人惊讶的是,在这项工作中,我们表明,在ANN到SNN的转换过程中,一种适当的方法来校准参数可以带来显着的改善。我们介绍SNN校准,一种廉价但非常有效的方法,利用预先训练的人工神经网络(ANN)内的知识。从理论上分析了转换误差及其在各层间的传播,提出了逐层校正误差的校正算法。校准只需要少量的训练数据和几分钟就可以完成。此外,我们的校准算法可以在大规模ImageNet数据集上生成具有最先进架构的SNN,包括MobileNet和RegNet。大量的实验证明了该算法的有效性和效率。例如,在ImageNet上转换MobileNet时,与基线相比,我们的高级管道可以提高高达69%的top-1准确率。代码发布于https://github.com/yhhhli/SNN_Calibration。

英文摘要:

Spiking Neural Network (SNN) has been recognized as one of the next generation of neural networks. Conventionally, SNN can be converted from a pre-trained ANN by only replacing the ReLU activation to spike activation while keeping the parameters intact. Perhaps surprisingly, in this work we show that a proper way to calibrate the parameters during the conversion of ANN to SNN can bring significant improvements. We introduce SNN Calibration, a cheap but extraordinarily effective method by leveraging the knowledge within a pre-trained Artificial Neural Network (ANN). Starting by analyzing the conversion error and its propagation through layers theoretically, we propose the calibration algorithm that can correct the error layer-by-layer. The calibration only takes a handful number of training data and several minutes to finish. Moreover, our calibration algorithm can produce SNN with state-of-the-art architecture on the large-scale ImageNet dataset, including MobileNet and RegNet. Extensive experiments demonstrate the effectiveness and efficiency of our algorithm. For example, our advanced pipeline can increase up to 69% top-1 accuracy when converting MobileNet on ImageNet compared to baselines. Codes are released at https://github.com/yhhhli/SNN_Calibration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑