Despite the rapid progress of neuromorphic computing, the inadequate depth and the resulting insufficient representation power of spiking neural networks (SNNs) severely restrict their application scope in practice. Residual learning and shortcuts have been evidenced as an important approach for training deep neural networks, but rarely did previous work assess their applicability to the characteristics of spike-based communication and spatiotemporal dynamics. This negligence leads to impeded information flow and the accompanying degradation problem. In this paper, we identify the crux and then propose a novel residual block for SNNs, which is able to significantly extend the depth of directly trained SNNs, e.g., up to 482 layers on CIFAR-10 and 104 layers on ImageNet, without observing any slight degradation problem. We validate the effectiveness of our methods on both frame-based and neuromorphic datasets, and our SRM-ResNet104 achieves a superior result of 76.02% accuracy on ImageNet, the first time in the domain of directly trained SNNs. The great energy efficiency is estimated and the resulting networks need on average only one spike per neuron for classifying an input sample. We believe our powerful and scalable modeling will provide a strong support for further exploration of SNNs.
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Peng Cheng Laboratory（鹏城实验室）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
A feedback control optimizer for online and hardware-aware training of Spiking Neural Networks
Institute of Neuroinformatics, University of Zürich and ETH Zürich（神经信息学研究所，苏黎世大学和苏黎世联邦理工学院）；ETH Zürich, Zürich, Switzerland（苏黎世联邦理工学院，苏黎世，瑞士）；Digital Society Initiative, University of Zurich, Switzerland（数字社会倡议，苏黎世大学，瑞士）；AI Center, ETH, Zürich, Switzerland（人工智能中心，苏黎世联邦理工学院，苏黎世，瑞士）
CogniSNN: Enabling Neuron-Expandability, Pathway-Reusability, and Dynamic-Configurability with Random Graph Architectures in Spiking Neural Networks
School of Software（软件学院）；Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Department of Computing（计算学院）；The Hong Kong Polytechnic University（香港理工大学）；Department of Data Science and AI（数据科学与人工智能学院）；Monash University（墨尔本大学）
Energy-Aware Spike Budgeting for Continual Learning in Spiking Neural Networks for Neuromorphic Vision
Dept. of Electrical and Electronics Engineering, University of Liberal Arts Bangladesh（乌拉尔大学 bangladesh 电子与电气工程系）；Dept. of Electrical Engineering, Pennsylvania State University（宾夕法尼亚州立大学电气工程系）
Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；ZJU-UIUC Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院）
Tsinghua University（清华大学）；Technische Universität Graz（格拉茨工业大学）；Xi’an Jiaotong University（西安交通大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Despite the rapid progress of neuromorphic computing, inadequate capacity and insufficient representation power of spiking neural networks (SNNs) severely restrict their application scope in practice. Residual learning and shortcuts have been evidenced as an important approach for training deep neural networks, but rarely did previous work assess their applicability to the characteristics of spike-based communication and spatiotemporal dynamics. In this paper, we first identify that this negligence leads to impeded information flow and the accompanying degradation problem in previous residual …
Much of studies on neural computation are based on network models of static neurons that produce analog output, despite the fact that information processing in the brain is predominantly carried out by dynamic neurons that produce discrete pulses called spikes. Research in spike-based computation has been impeded by the lack of efficient supervised learning algorithm for spiking networks. Here, we present a gradient descent method for optimizing spiking network models by introducing a differentiable formulation of spiking networks and deriving the exact gradient calculation. For demonstration,…
Stanford University（斯坦福大学）；University of Oxford（牛津大学）
A vast majority of computation in the brain is performed by spiking neural networks. Despite the ubiquity of such spiking, we currently lack an understanding of how biological spiking neural circuits learn and compute in-vivo, as well as how we can instantiate such capabilities in artificial spiking circuits in-silico. Here we revisit the problem of supervised learning in temporally coding multi-layer spiking neural networks. First, by using a surrogate gradient approach, we derive SuperSpike, a nonlinear voltage-based three factor learning rule capable of training multi-layer networks of dete…
Stochastic Quantum Spiking Neural Networks with Quantum Memory and Local Learning
Department of Engineering, King’s College London（伦敦国王学院工程系）；Institute for Intelligent Networked Systems, Northeastern University London（伦敦东北大学智能网络系统研究所）
Brain-inspired computing proposes a set of algorithmic principles that hold promise for advancing artificial intelligence. They endow systems with self learning capabilities, efficient energy usage, and high storage capacity. A core concept that lies at the heart of brain computation is sequence learning and prediction. This form of computation is essential for almost all our daily tasks such as movement generation, perception, and language. Understanding how the brain performs such a computation is not only important to advance neuroscience but also to pave the way to new technological brain-…
Event-driven eligibility propagation in large sparse networks: efficiency shaped by biological realism
Institute for Advanced Simulation 6 (IAS-6), Jülich Research Centre（先进模拟研究所第六所（IAS-6），于利希研究中心）；RWTH Aachen University（亚琛工业大学）；University of Wuppertal（伍珀塔尔大学）；University of Bern（伯尔尼大学）；Norwegian University of Life Sciences（挪威生命科学大学）；Käte Hamburger Kolleg, RWTH Aachen University（凯特·汉伯格学院，亚琛工业大学）；Institute of Zoology, University of Cologne（科隆大学动物学研究所）
