Spiking Neural Networks (SNNs) have been recently integrated into Transformer architectures due to their potential to reduce computational demands and to improve power efficiency. Yet, the implementation of the attention mechanism using spiking signals on general-purpose computing platforms remains inefficient. In this paper, we propose a novel framework leveraging stochastic computing (SC) to effectively execute the dot-product attention for SNN-based Transformers. We demonstrate that our approach can achieve high classification accuracy ($83.53\%$) on CIFAR-10 within 10 time steps, which is comparable to the performance of a baseline artificial neural network implementation ($83.66\%$). We estimate that the proposed SC approach can lead to over $6.3\times$ reduction in computing energy and $1.7\times$ reduction in memory access costs for a digital CMOS-based ASIC design. We experimentally validate our stochastic attention block design through an FPGA implementation, which is shown to achieve $48\times$ lower latency as compared to a GPU implementation, while consuming $15\times$ less power.
NeuroNAS: Enhancing Efficiency of Neuromorphic In-Memory Computing for Intelligent Mobile Agents through Hardware-Aware Spiking Neural Architecture Search
EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis
School of Computing, University of Georgia（佐治亚大学计算学院）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Georgia（佐治亚大学）；Department of Physics and Astronomy（物理与天文学系）
SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models
Southern University of Science and Technology（南方科技大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）；Peking University（北京大学）；City University of Hong Kong（香港城市大学）；Pengcheng Laboratory（鹏城实验室）
DeepFire2: A Convolutional Spiking Neural Network Accelerator on FPGAs
Institute of High Performance Computing, Agency for Science Technology and Research (A*STAR)（科学、技术与研究机构（A*STAR）高性能计算研究院）；National University of Singapore（新加坡国立大学）
