脉冲神经网络中基于权重初始化的深度活动传播
Deep activity propagation via weight initialization in spiking neural networks
AI总结：
针对深度脉冲神经网络训练中脉冲消失的难题，提出一种考虑量化操作的专用权重初始化方法，理论与实验验证其可实现无脉冲损失的深度活动传播，提升MNIST数据集上的训练精度与收敛速度，且对超参数变化鲁棒。
摘要：
脉冲神经网络（Spiking Neural Networks, SNNs）与神经形态计算具备稀疏性、超低功耗等生物启发式优势，是传统网络的有力替代方案。然而，从零开始训练深度SNN仍是一项挑战：SNN通过将实值膜电位量化为二值脉冲来处理和传输信息，这会导致信息损失与深层网络中的脉冲消失，阻碍有效训练。尽管权重初始化对深度神经网络训练至关重要，但深度SNN的有效初始状态尚未得到充分研究。现有针对传统人工神经网络（ANNs）设计的权重初始化方法常被直接应用于SNN，却未考虑其独特的计算特性。\n本研究推导了一种专门适配SNN、考虑量化操作的最优权重初始化方法。我们从理论上证明，与标准方法不同，该方法可实现深度SNN中的活动传播且无脉冲损失。我们在多达100层、跨多个时间步的SNN数值模拟中验证了这一特性，并深入分析了准确应用理论结论所需的数值条件，涉及层宽度与神经元超参数。此外，我们在MNIST数据集上的实验表明，采用所提权重初始化方案可实现更高精度与更快收敛速度。最后，我们证明该新型权重初始化方法对多种网络与神经元超参数的变化具有鲁棒性。
英文摘要：
Spiking Neural Networks (SNNs) and neuromorphic computing offer bio-inspired advantages such as sparsity and ultra-low power consumption, providing a promising alternative to conventional networks. However, training deep SNNs from scratch remains a challenge, as SNNs process and transmit information by quantizing the real-valued membrane potentials into binary spikes. This can lead to information loss and vanishing spikes in deeper layers, impeding effective training. While weight initialization is known to be critical for training deep neural networks, what constitutes an effective initial state for a deep SNN is not well-understood. Existing weight initialization methods designed for conventional networks (ANNs) are often applied to SNNs without accounting for their distinct computational properties. In this work we derive an optimal weight initialization method specifically tailored for SNNs, taking into account the quantization operation. We show theoretically that, unlike standard approaches, this method enables the propagation of activity in deep SNNs without loss of spikes. We demonstrate this behavior in numerical simulations of SNNs with up to 100 layers across multiple time steps. We present an in-depth analysis of the numerical conditions, regarding layer width and neuron hyperparameters, which are necessary to accurately apply our theoretical findings. Furthermore, our experiments on MNIST demonstrate higher accuracy and faster convergence when using the proposed weight initialization scheme. Finally, we show that the newly introduced weight initialization is robust against variations in several network and neuron hyperparameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
