SpikingSSMs：利用稀疏且并行的脉冲状态空间模型学习长序列
SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models
AI总结：
本研究提出SpikingSSMs，通过分层集成神经元动态与SSM模块实现稀疏突触计算，并设计轻量级替代动态网络解决并行训练冲突，在保持90%稀疏度的同时于长序列任务上取得SOTA性能，且以更小模型规模超越现有脉冲大语言模型。
摘要：
作为低能耗网络，脉冲神经网络（SNNs）在过去几十年中受到了广泛关注。尽管SNNs在视觉任务中与人工神经网络（ANNs）的竞争力日益增强，但尽管其具有内在的时间动态特性，却很少用于长序列任务。在本研究中，我们利用状态空间模型（SSMs）的序列学习能力，开发了用于长序列学习的脉冲状态空间模型（SpikingSSMs）。受树突神经元结构的启发，我们将神经元动态与原始SSM模块分层集成，同时实现了稀疏突触计算。此外，为了解决事件驱动的神经元动态与并行计算之间的冲突，我们提出了一个轻量级的替代动态网络，该网络能准确预测重置后的膜电位，并兼容可学习阈值，与传统的迭代方法相比，实现了数个数量级的训练速度加速。在长程竞技场基准任务上，SpikingSSM实现了与最先进SSMs相媲美的性能，同时平均实现了90%的网络稀疏度。在语言建模方面，我们的网络在WikiText-103数据集上以仅三分之一的模型大小显著超越了现有的脉冲大型语言模型（spikingLLMs），展示了其作为低计算成本LLMs骨干架构的潜力。
英文摘要：
Known as low energy consumption networks, spiking neural networks (SNNs) have gained a lot of attention within the past decades. While SNNs are increasing competitive with artificial neural networks (ANNs) for vision tasks, they are rarely used for long sequence tasks, despite their intrinsic temporal dynamics. In this work, we develop spiking state space models (SpikingSSMs) for long sequence learning by leveraging on the sequence learning abilities of state space models (SSMs). Inspired by dendritic neuron structure, we hierarchically integrate neuronal dynamics with the original SSM block, meanwhile realizing sparse synaptic computation. Furthermore, to solve the conflict of event-driven neuronal dynamics with parallel computing, we propose a light-weight surrogate dynamic network which accurately predicts the after-reset membrane potential and compatible to learnable thresholds, enabling orders of acceleration in training speed compared with conventional iterative methods. On the long range arena benchmark task, SpikingSSM achieves competitive performance to state-of-the-art SSMs meanwhile realizing on average 90\% of network sparsity. On language modeling, our network significantly surpasses existing spiking large language models (spikingLLMs) on the WikiText-103 dataset with only a third of the model size, demonstrating its potential as backbone architecture for low computation cost LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
