关于脉冲神经网络二阶优化的研究
On the second-order optimization for spiking neural networks
- Centre for Secure Information Technologies（安全信息技术中心）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
针对脉冲神经网络训练中尖锐损失景观与曲率信息利用不足的问题，提出二阶优化方法SpiKFAX，通过Kronecker因子近似Fisher信息矩阵，在多个架构与数据集上提升测试精度与训练稳定性。
中文摘要 AI 辅助
脉冲神经网络（SNNs）通过利用稀疏的二值脉冲和事件驱动计算，为传统神经网络提供了一种节能的替代方案。然而，SNNs的训练仍然具有挑战性，因为脉冲激活产生了阻碍训练的尖锐损失景观，而诸如Adam系列等对角曲率优化器可能无法捕捉这一几何特性。基于曲率的优化方法扩展到SNNs进一步受到其底层动态的稀疏性、离散性和时间递归性的阻碍。为了解决这些限制，我们提出了SpiKFAX，一种二阶优化方法，该方法构建了专门适应SNNs结构的Fisher信息矩阵的计算上可行的Kronecker因子近似。在五个架构和七个数据集上的实证评估表明，与其他流行的优化器相比，SpiKFAX在测试准确性和训练稳定性方面持续带来改进。
英文摘要
Spiking Neural Networks (SNNs) offer an energy-efficient alternative to conventional neural networks by exploiting sparse, binary spikes, and event-driven computation. However, the training of SNNs remains challenging, as spiking activations create a sharp loss landscape that hinders training, and diagonal-curvature optimizers such as the Adam family may fail to capture this geometry. The extension of curvature-based optimization methods to SNNs is further complicated by the sparse, discrete, and temporally recurrent nature of their underlying dynamics. To address these limitations, we propose SpiKFAX, a second-order optimization method that formulates a computationally tractable, Kronecker-factored approximation of the Fisher information matrix specifically adapted to the structure of SNNs. Empirical evaluation across five architectures and seven datasets demonstrates that SpiKFAX consistently yields improvements in test accuracy and training stability relative to other popular optimizers.
