arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.04486cs.CV
收藏

抄近路返回:缓解脉冲神经网络训练中的梯度消失问题

Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks

Intelligent Science & Technology Academy of CASIC(中国航天科工集团智能科学与技术研究院);School of Computer Science, Peking University(北京大学计算机学院)

Yufei Guo, Yuanpei Chen, Zecheng Hao, Weihang Peng, Zhou Jie, Yuhan Zhang, Xiaode Liu, Zhe Ma

AI总结:

本文针对脉冲神经网络训练中的梯度消失问题,提出捷径反向传播方法将梯度直接传递到浅层,并结合进化训练框架动态调整平衡系数,在多个数据集和网络结构上取得了优于现有方法的性能。

摘要:

脉冲神经网络(SNN)是一种受生物启发的神经网络基础设施,近年来引起了广泛关注。它利用二值脉冲激活来传递信息,从而用加法替代乘法,实现了高能效。然而,由于发放脉冲过程的梯度未定义,直接训练SNN面临挑战。尽管先前的工作采用了各种替代梯度训练方法,在反向传播过程中使用替代函数来替换发放过程,但这些方法忽略了一个内在问题:梯度消失。为了解决这一问题,我们在本文中提出了一种捷径反向传播方法,主张将梯度直接从损失传递到浅层。这使我们能够直接将梯度呈现给浅层,从而显著缓解梯度消失问题。此外,该方法在推理阶段不会引入任何负担。为了在最终精度和训练简易性之间取得平衡,我们还提出了一种进化训练框架,并通过引入一个随训练轮次动态变化的平衡系数来实现它,这进一步提升了网络的性能。在静态和动态数据集上使用几种流行的网络结构进行的大量实验表明,我们的方法始终优于最先进的方法。

英文摘要:

The Spiking Neural Network (SNN) is a biologically inspired neural network infrastructure that has recently garnered significant attention. It utilizes binary spike activations to transmit information, thereby replacing multiplications with additions and resulting in high energy efficiency. However, training an SNN directly poses a challenge due to the undefined gradient of the firing spike process. Although prior works have employed various surrogate gradient training methods that use an alternative function to replace the firing process during back-propagation, these approaches ignore an intrinsic problem: gradient vanishing. To address this issue, we propose a shortcut back-propagation method in our paper, which advocates for transmitting the gradient directly from the loss to the shallow layers. This enables us to present the gradient to the shallow layers directly, thereby significantly mitigating the gradient vanishing problem. Additionally, this method does not introduce any burden during the inference phase. To strike a balance between final accuracy and ease of training, we also propose an evolutionary training framework and implement it by inducing a balance coefficient that dynamically changes with the training epoch, which further improves the network's performance. Extensive experiments conducted over static and dynamic datasets using several popular network structures reveal that our method consistently outperforms state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑