Spiking neural networks are efficient computation models for low-power environments. Spike-based BP algorithms and ANN-to-SNN (ANN2SNN) conversions are successful techniques for SNN training. Nevertheless, the spike-base BP training is slow and requires large memory costs. Though ANN2NN provides a low-cost way to train SNNs, it requires many inference steps to mimic the well-trained ANN for good performance. In this paper, we propose a SNN-to-ANN (SNN2ANN) framework to train the SNN in a fast and memory-efficient way. The SNN2ANN consists of 2 components: a) a weight sharing architecture between ANN and SNN and b) spiking mapping units. Firstly, the architecture trains the weight-sharing parameters on the ANN branch, resulting in fast training and low memory costs for SNN. Secondly, the spiking mapping units ensure that the activation values of the ANN are the spiking features. As a result, the classification error of the SNN can be optimized by training the ANN branch. Besides, we design an adaptive threshold adjustment (ATA) algorithm to address the noisy spike problem. Experiment results show that our SNN2ANN-based models perform well on the benchmark datasets (CIFAR10, CIFAR100, and Tiny-ImageNet). Moreover, the SNN2ANN can achieve comparable accuracy under 0.625x time steps, 0.377x training time, 0.27x GPU memory costs, and 0.33x spike activities of the Spike-based BP model.
Recent advances in event-based neuromorphic systems have resulted in significant interest in the use and development of spiking neural networks (SNNs). However, the non-differentiable nature of spiking neurons makes SNNs incompatible with conventional backpropagation techniques. In spite of the significant progress made in training conventional deep neural networks (DNNs), training methods for SNNs still remain relatively poorly understood. In this paper, we present a novel framework for training recurrent SNNs. Analogous to the benefits presented by recurrent neural networks (RNNs) in learnin…
State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院工业控制技术国家重点实验室）；Key Laboratory of Collaborative sensing and autonomous unmanned systems of Zhejiang Province（浙江省协同感知与自主无人系统重点实验室）
Graz University of Technology（格拉茨技术大学）；Silicon Austria Labs（奥地利硅谷实验室）
Memory is a key component of biological neural systems that enables the retention of information over a huge range of temporal scales, ranging from hundreds of milliseconds up to years. While Hebbian plasticity is believed to play a pivotal role in biological memory, it has so far been analyzed mostly in the context of pattern completion and unsupervised learning. Here, we propose that Hebbian plasticity is fundamental for computations in biological neural systems. We introduce a novel spiking neural network architecture that is enriched by Hebbian synaptic plasticity. We show that Hebbian enr…
Neuromorphic engineering concentrates the efforts of a large number of researchers due to its great potential as a field of research, in a search for the exploitation of the advantages of the biological nervous system and the brain as a whole for the design of more efficient and real-time capable applications. For the development of applications as close to biology as possible, Spiking Neural Networks (SNNs) are used, considered biologically-plausible and that form the third generation of Artificial Neural Networks (ANNs). Since some SNN-based applications may need to store data in order to us…
Tufts University（塔夫茨大学）；University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）
A common view in the neuroscience community is that memory is encoded in the connection strength between neurons. This perception led artificial neural network models to focus on connection weights as the key variables to modulate learning. In this paper, we present a prototype for weightless spiking neural networks that can perform a simple classification task. The memory in this network is stored in the timing between neurons, rather than the strength of the connection, and is trained using a Hebbian Spike Timing Dependent Plasticity (STDP), which modulates the delays of the connection.
Stochastic Quantum Spiking Neural Networks with Quantum Memory and Local Learning
Department of Engineering, King’s College London（伦敦国王学院工程系）；Institute for Intelligent Networked Systems, Northeastern University London（伦敦东北大学智能网络系统研究所）
University of California San Diego（加利福尼亚大学圣地亚哥分校）；University of California Los Angeles（加利福尼亚大学洛杉矶分校）；University of Connecticut（康涅狄格大学）；Daegu Gyeongbuk Institute of Science and Technology（大邱庆北科学技术院）；University of California Irvine（加利福尼亚大学欧文分校）
Recently, brain-inspired computing models have shown great potential to outperform today's deep learning solutions in terms of robustness and energy efficiency. Particularly, Spiking Neural Networks (SNNs) and HyperDimensional Computing (HDC) have shown promising results in enabling efficient and robust cognitive learning. Despite the success, these two brain-inspired models have different strengths. While SNN mimics the physical properties of the human brain, HDC models the brain on a more abstract and functional level. Their design philosophies demonstrate complementary patterns that motivat…
We present a Spiking Neural Network (SNN) based Sparse Distributed Memory (SDM) implemented on the Nengo framework. We have based our work on previous work by Furber et al, 2004, implementing SDM using N-of-M codes. As an integral part of the SDM design, we have implemented Correlation Matrix Memory (CMM) using SNN on Nengo. Our SNN implementation uses Leaky Integrate and Fire (LIF) spiking neuron models on Nengo. Our objective is to understand how well SNN-based SDMs perform in comparison to conventional SDMs. Towards this, we have simulated both conventional and SNN-based SDM and CMM on Neng…
Spiking recurrent neural networks (RNNs) are a promising tool for solving a wide variety of complex cognitive and motor tasks, due to their rich temporal dynamics and sparse processing. However training spiking RNNs on dedicated neuromorphic hardware is still an open challenge. This is due mainly to the lack of local, hardware-friendly learning mechanisms that can solve the temporal credit assignment problem and ensure stable network dynamics, even when the weight resolution is limited. These challenges are further accentuated, if one resorts to using memristive devices for in-memory computing…
The event-driven and sparse nature of communication between spiking neurons in the brain holds great promise for flexible and energy-efficient AI. Recent advances in learning algorithms have demonstrated that recurrent networks of spiking neurons can be effectively trained to achieve competitive performance compared to standard recurrent neural networks. Still, as these learning algorithms use error-backpropagation through time (BPTT), they suffer from high memory requirements, are slow to train, and are incompatible with online learning. This limits the application of these learning algorithm…
Ghent University（根特大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）
The information of spiking neural networks (SNNs) are propagated between the adjacent biological neuron by spikes, which provides a computing paradigm with the promise of simulating the human brain. Recent studies have found that the time delay of neurons plays an important role in the learning process. Therefore, configuring the precise timing of the spike is a promising direction for understanding and improving the transmission process of temporal information in SNNs. However, most of the existing learning methods for spiking neurons are focusing on the adjustment of synaptic weight, while v…
A draft memory model (DM) for neural networks with spike propagation delay (SNNwD) is described. Novelty in this approach are that the DM learns immediately, with stimuli presented once, without synaptic weight changes, and without external learning algorithm. Basal on this model is to trap spikes within neural loops. In order to construct the DM we developed two functional blocks, also described herein. The decoder block receives input from a single spikes source and connect it to one among many outputs. The selector block operates in the opposite direction, receiving many spikes sources and …
Technische Universität Wien (TU Wien)（维也纳工业大学）；New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校）
Larger Spiking Neural Network (SNN) models are typically favorable as they can offer higher accuracy. However, employing such models on the resource- and energy-constrained embedded platforms is inefficient. Towards this, we present a tinySNN framework that optimizes the memory and energy requirements of SNN processing in both the training and inference phases, while keeping the accuracy high. It is achieved by reducing the SNN operations, improving the learning quality, quantizing the SNN parameters, and selecting the appropriate SNN model. Furthermore, our tinySNN quantizes different SNN par…
Spike Timing Dependent Plasticity is form of learning that has been demonstrated in real cortical tissue, but attempts to use it for artificial systems have not produced good results. This paper seeks to remedy this with two significant advances. The first is the development a simple stochastic learning rule called cyclic STDP that can extract patterns encoded in the precise spiking times of a group of neurons. We show that a population of neurons endowed with this learning rule can act as an effective short-term associative memory, storing and reliably recalling a large set of pattern associa…
Spiking neural networks, also often referred to as the third generation of neural networks, carry the potential for a massive reduction in memory and energy consumption over traditional, second-generation neural networks. Inspired by the undisputed efficiency of the human brain, they introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. To open the pathway toward engineering applications, we introduce this exciting technology in the context of continuum mechanics. However, the nature of spiking neural networks poses a challenge for regressio…
The spiking neural networks (SNNs) are considered as one of the most promising artificial neural networks due to their energy efficient computing capability. Recently, conversion of a trained deep neural network to an SNN has improved the accuracy of deep SNNs. However, most of the previous studies have not achieved satisfactory results in terms of inference speed and energy efficiency. In this paper, we propose a fast and energy-efficient information transmission method with burst spikes and hybrid neural coding scheme in deep SNNs. Our experimental results showed the proposed methods can imp…
Spiking neural networks (SNN) are artificial computational models that have been inspired by the brain's ability to naturally encode and process information in the time domain. The added temporal dimension is believed to render them more computationally efficient than the conventional artificial neural networks, though their full computational capabilities are yet to be explored. Recently, computational memory architectures based on non-volatile memory crossbar arrays have shown great promise to implement parallel computations in artificial and spiking neural networks. In this work, we experim…
Technische Universität Wien (TU Wien)（维也纳工业大学）；New York University Abu Dhabi (NYU AD)（纽约大学阿布扎比分校）
Spiking Neural Networks (SNNs) are gaining interest due to their event-driven processing which potentially consumes low power/energy computations in hardware platforms, while offering unsupervised learning capability due to the spike-timing-dependent plasticity (STDP) rule. However, state-of-the-art SNNs require a large memory footprint to achieve high accuracy, thereby making them difficult to be deployed on embedded systems, for instance on battery-powered mobile devices and IoT Edge nodes. Towards this, we propose FSpiNN, an optimization framework for obtaining memory- and energy-efficient …
Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）
Biological neurons communicate with a sparing exchange of pulses - spikes. It is an open question how real spiking neurons produce the kind of powerful neural computation that is possible with deep artificial neural networks, using only so very few spikes to communicate. Building on recent insights in neuroscience, we present an Adapting Spiking Neural Network (ASNN) based on adaptive spiking neurons. These spiking neurons efficiently encode information in spike-trains using a form of Asynchronous Pulsed Sigma-Delta coding while homeostatically optimizing their firing rate. In the proposed par…
VQ4SNN: Vector Quantization for Memory-Efficient FPGA Spiking Neural Networks
Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece（德米特里乌斯特拉基亚大学集成电路实验室，电气与计算机工程，特拉基亚大学（DUTH），希腊）
Neural Dynamics Self-Attention for Spiking Transformers
University of Electronic Science and Technology of China（电子科技大学）；Shenzhen Loop Area Institute（深圳河套学院）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
Spiking Neural Networks (SNNs), as bio-inspired energy-efficient neural networks, have attracted great attentions from researchers and industry. The most efficient way to train deep SNNs is through ANN-SNN conversion. However, the conversion usually suffers from accuracy loss and long inference time, which impede the practical application of SNN. In this paper, we theoretically analyze ANN-SNN conversion and derive sufficient conditions of the optimal conversion. To better correlate ANN-SNN and get greater accuracy, we propose Rate Norm Layer to replace the ReLU activation function in source A…
While classical neural networks take a position of a leading method in the machine learning community, spiking neuromorphic systems bring attention and large projects in neuroscience. Spiking neural networks were shown to be able to substitute networks of classical neurons in applied tasks. This work explores recent hardware designs focusing on perspective applications (like convolutional neural networks) for both neuron types from the energy efficiency side to analyse whether there is a possibility for spiking neuromorphic hardware to grow up for a wider use. Our comparison shows that spiking…
A robust real-world deployment of autonomous edge devices requires on-chip adaptation to user-, environment- and task-induced variability. Due to on-chip memory constraints, prior learning devices were limited to static stimuli with no temporal contents. We propose a 0.45-mm$^2$ spiking RNN processor enabling task-agnostic online learning over seconds, which we demonstrate for navigation, gesture recognition, and keyword spotting within a 0.8-% memory overhead and a <150-$μ$W training power budget.
Evaluation of spiking neural networks requires fetching a large number of synaptic weights to update postsynaptic neurons. This limits parallelism and becomes a bottleneck for hardware. We present an approach for spike propagation based on a probabilistic interpretation of weights, thus reducing memory accesses and updates. We study the effects of introducing randomness into the spike processing, and show on benchmark networks that this can be done with minimal impact on the recognition accuracy. We present an architecture and the trade-offs in accuracy on fully connected and convolutional net…
SpikeNAS: A Fast Memory-Aware Neural Architecture Search Framework for Spiking Neural Network-based Embedded AI Systems
eBrain Lab, Division of Engineering, New York University (NYU) Abu Dhabi, United Arab Emirates（eBrain实验室，工程系，纽约大学（NYU）阿布扎赫德分校）；New York University (NYU) Abu Dhabi, United Arab Emirates（纽约大学（NYU）阿布扎赫德分校）
Technische Universität Wien (TU Wien)（维也纳工业大学（TU Wien））；New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校（NYUAD））
A prominent technique for reducing the memory footprint of Spiking Neural Networks (SNNs) without decreasing the accuracy significantly is quantization. However, the state-of-the-art only focus on employing the weight quantization directly from a specific quantization scheme, i.e., either the post-training quantization (PTQ) or the in-training quantization (ITQ), and do not consider (1) quantizing other SNN parameters (e.g., neuron membrane potential), (2) exploring different combinations of quantization approaches (i.e., quantization schemes, precision levels, and rounding schemes), and (3) s…
Spiking Neural Networks (SNNs) have recently gained popularity as machine learning models for on-device edge intelligence for applications such as mobile healthcare management and natural language processing due to their low power profile. In such highly personalized use cases, it is important for the model to be able to adapt to the unique features of an individual with only a minimal amount of training data. Meta-learning has been proposed as a way to train models that are geared towards quick adaptation to new tasks. The few existing meta-learning solutions for SNNs operate offline and requ…
Adversarially Robust Spiking Neural Networks with Sparse Connectivity
Institute of Machine Learning and Neural Computation, Graz University of Technology, Austria（机器学习与神经计算研究所，格拉茨技术大学）；TU Graz - SAL Dependable Embedded Systems Lab, Silicon Austria Labs, Austria（格拉茨大学 - SAL 可靠嵌入式系统实验室，Silicon Austria Labs）；Chair of Cyber-Physical-Systems, Montanuniversität Leoben, Austria（智能物理系统系，蒙特安大学）
There is an increasing interest in emulating Spiking Neural Networks (SNNs) on neuromorphic computing devices due to their low energy consumption. Recent advances have allowed training SNNs to a point where they start to compete with traditional Artificial Neural Networks (ANNs) in terms of accuracy, while at the same time being energy efficient when run on neuromorphic hardware. However, the process of training SNNs is still based on dense tensor operations originally developed for ANNs which do not leverage the spatiotemporally sparse nature of SNNs. We present here the first sparse SNN back…
The increasing need for compact and low-power computing solutions for machine learning applications has triggered significant interest in energy-efficient neuromorphic systems. However, most of these architectures rely on spiking neural networks, which typically perform poorly compared to their non-spiking counterparts in terms of accuracy. In this paper, we propose a new adaptive spiking neuron model that can be abstracted as a low-pass filter. This abstraction enables faster and better training of spiking networks using back-propagation, without simulating spikes. We show that this model dra…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of California–Irvine（加州大学尔湾分校）
Directly training spiking neural networks (SNNs) has remained challenging due to complex neural dynamics and intrinsic non-differentiability in firing functions. The well-known backpropagation through time (BPTT) algorithm proposed to train SNNs suffers from large memory footprint and prohibits backward and update unlocking, making it impossible to exploit the potential of locally-supervised training methods. This work proposes an efficient and direct training algorithm for SNNs that integrates a locally-supervised training method with a temporally-truncated BPTT algorithm. The proposed algori…
Yale University（耶鲁大学）；Arizona State University（亚利桑那州立大学）
SNNs are an active research domain towards energy efficient machine intelligence. Compared to conventional ANNs, SNNs use temporal spike data and bio-plausible neuronal activation functions such as Leaky-Integrate Fire/Integrate Fire (LIF/IF) for data processing. However, SNNs incur significant dot-product operations causing high memory and computation overhead in standard von-Neumann computing platforms. Today, In-Memory Computing (IMC) architectures have been proposed to alleviate the "memory-wall bottleneck" prevalent in von-Neumann architectures. Although recent works have proposed IMC-bas…
NeuroNAS: Enhancing Efficiency of Neuromorphic In-Memory Computing for Intelligent Mobile Agents through Hardware-Aware Spiking Neural Architecture Search
PrimeSVT: An Automated Memory-aware Pruning Framework with Prioritized Compression Policy for Spiking Vision Transformers
eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi（eBRAIN实验室，工程系，纽约大学（NYU）阿布扎克分校）；New York University (NYU) Abu Dhabi, United Arab Emirates (UAE)（纽约大学（NYU）阿布扎克分校，阿拉伯联合酋长国（UAE））
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）；Binghamton University, The State University of New York（纽约州立大学宾汉姆顿分校）
Accurate real-time object recognition from sensory data has long been a crucial and challenging task for autonomous driving. Even though deep neural networks (DNNs) have been successfully applied in this area, most existing methods still heavily rely on the pre-processing of the pulse signals derived from LiDAR sensors, and therefore introduce additional computational overhead and considerable latency. In this paper, we propose an approach to address the object recognition problem directly with raw temporal pulses utilizing the spiking neural network (SNN). Being evaluated on various datasets …
