Depth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here we solved it using an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Network (SNN) with a slightly modified U-Net-like encoder-decoder architecture, that we named StereoSpike. More specifically, we used the Multi Vehicle Stereo Event Camera Dataset (MVSEC). It provides a depth ground-truth, which was used to train StereoSpike in a supervised manner, using surrogate gradient descent. We propose a novel readout paradigm to obtain a dense analog prediction -- the depth of each pixel -- from the spikes of the decoder. We demonstrate that this architecture generalizes very well, even better than its non-spiking counterparts, leading to state-of-the-art test accuracy. To the best of our knowledge, it is the first time that such a large-scale regression problem is solved by a fully spiking network. Finally, we show that low firing rates (<10%) can be obtained via regularization, with a minimal cost in accuracy. This means that StereoSpike could be efficiently implemented on neuromorphic chips, opening the door for low power and real time embedded systems.
Despite the rapid progress of neuromorphic computing, the inadequate depth and the resulting insufficient representation power of spiking neural networks (SNNs) severely restrict their application scope in practice. Residual learning and shortcuts have been evidenced as an important approach for training deep neural networks, but rarely did previous work assess their applicability to the characteristics of spike-based communication and spatiotemporal dynamics. This negligence leads to impeded information flow and the accompanying degradation problem. In this paper, we identify the crux and the…
Spiking PointNet: Spiking Neural Networks for Point Clouds
Intelligent Science & Technology Academy of CASIC（中国航天科工集团智能科技研究院）；Scientific Research Laboratory of Aerospace Intelligent Systems and Technology（航天智能系统技术科研实验室）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
Spiking Neural Networks (SNNs) are more biologically plausible and computationally efficient. Therefore, SNNs have the natural advantage of drawing the sparse structural plasticity of brain development to alleviate the energy problems of deep neural networks caused by their complex and fixed structures. However, previous SNNs compression works are lack of in-depth inspiration from the brain development plasticity mechanism. This paper proposed a novel method for the adaptive structural development of SNN (SD-SNN), introducing dendritic spine plasticity-based synaptic constraint, neuronal pruni…
Deep Directly-Trained Spiking Neural Networks for Object Detection
University of Chinese Academy of Sciences（中国科学院大学）；Xi’an Jiaotong University（西安交通大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua University（清华大学）；Peking University（北京大学）；Huawei Technologies Co Ltd.（华为技术有限公司）
Institute of Neuroinformatics, University of Zurich and ETH Zurich（苏黎世大学与苏黎世联邦理工学院神经信息学研究所）
The stereo-matching problem, i.e., matching corresponding features in two different views to reconstruct depth, is efficiently solved in biology. Yet, it remains the computational bottleneck for classical machine vision approaches. By exploiting the properties of event cameras, recently proposed Spiking Neural Network (SNN) architectures for stereo vision have the potential of simplifying the stereo-matching problem. Several solutions that combine event cameras with spike-based neuromorphic processors already exist. However, they are either simulated on digital hardware or tested on simplified…
LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks
Zhejiang University（浙江大学）；Zhejiang University School of Medicine（浙江大学医学院）；Affiliated Mental Health Center & Hangzhou Seventh People’s Hospital（附属精神卫生中心&杭州市第七人民医院）
Adaptive structure evolution and biologically plausible synaptic plasticity for recurrent spiking neural networks
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
Evolving Efficient Genetic Encoding for Deep Spiking Neural Networks
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Center for Long-term Artificial Intelligence（长期人工智能中心）；Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Chinese Academy of Sciences（中国科学院脑认知与脑智能技术重点实验室）
Zhejiang University（浙江大学）；Tsinghua University（清华大学）；Xi’an Jiaotong University（西安交通大学）；Huawei Technologies Co. Ltd（华为技术有限公司）；Chinese Academy of Sciences（中国科学院）
Event cameras are considered to have great potential for computer vision and robotics applications because of their high temporal resolution and low power consumption characteristics. However, the event stream output from event cameras has asynchronous, sparse characteristics that existing computer vision algorithms cannot handle. Spiking neural network is a novel event-based computational paradigm that is considered to be well suited for processing event camera tasks. However, direct training of deep SNNs suffers from degradation problems. This work addresses these problems by proposing a spi…
College of Computer Science and Technology（计算机科学与技术学院）；State Key Laboratory of Brain-machine Intelligence（脑机智能国家重点实验室）；Zhejiang University（浙江大学）；School of Software Technology（软件学院）；School of Computer Science（计算机学院）；Fudan University（复旦大学）；Institute of Data Science（数据科学研究院）；National University of Singapore（新加坡国立大学）；First Affiliated Hospital（第一附属医院）；Zhejiang University School of Medicine（浙江大学医学院）；Distributed Systems Group（分布式系统组）；Technische Universität Wien（维也纳技术大学）；ICREA Barcelona（巴塞罗那ICREA）；The University of Sydney（悉尼大学）
PrivSpike: Employing Homomorphic Encryption for Private Inference of Deep Spiking Neural Networks
STAM Center, Ira A. Fulton Schools of Engineering, Arizona State University（STAM中心、伊拉·A·福鲁顿工程学院、亚利桑那州立大学）；Faculty of Technical Sciences, University of Novi Sad（技术科学学院、诺维萨德大学）
SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike Streams
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理全国重点实验室）；Center on Frontiers of Computing Studies, School of Computer Science, Peking University（北京大学计算机学院计算前沿研究中心）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Engineering and Applied Science, University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院）
Neural networks have become the standard model for various computer vision tasks in automated driving including semantic segmentation, moving object detection, depth estimation, visual odometry, etc. The main flavors of neural networks which are used commonly are convolutional (CNN) and recurrent (RNN). In spite of rapid progress in embedded processors, power consumption and cost is still a bottleneck. Spiking Neural Networks (SNNs) are gradually progressing to achieve low-power event-driven hardware architecture which has a potential for high efficiency. In this paper, we explore the role of …
Over the past decade, deep neural networks (DNNs) have demonstrated remarkable performance in a variety of applications. As we try to solve more advanced problems, increasing demands for computing and power resources has become inevitable. Spiking neural networks (SNNs) have attracted widespread interest as the third-generation of neural networks due to their event-driven and low-powered nature. SNNs, however, are difficult to train, mainly owing to their complex dynamics of neurons and non-differentiable spike operations. Furthermore, their applications have been limited to relatively simple …
University of Strathclyde（思克莱德大学）；Air Force Research Laboratory（美国空军研究实验室）
Spiking neural networks (SNNs) have become an interesting alternative to conventional artificial neural networks (ANN) thanks to their temporal processing capabilities and energy efficient implementations in neuromorphic hardware. However the challenges involved in training SNNs have limited their performance in terms of accuracy and thus their applications. Improving learning algorithms and neural architectures for a more accurate feature extraction is therefore one of the current priorities in SNN research. In this paper we present a study on the key components of modern spiking architecture…
State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院工业控制技术国家重点实验室）；Key Laboratory of Collaborative sensing and autonomous unmanned systems of Zhejiang Province（浙江省协同感知与自主无人系统重点实验室）
FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks
Zhejiang University-University of Illinois Urbana Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院）；The College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）
Zhongnan University of Economics and Law（中南财经政法大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Yunnan University（云南大学）；National Institute of Natural Hazards, Ministry of Emergency Management of China（中国应急管理部国家自然灾害防治研究院）；Wuhan University（武汉大学）
Discretized Quadratic Integrate-and-Fire Neuron Model for Deep Spiking Neural Networks
STAM Center Arizona State University（STAM中心亚利桑那州立大学）；Center for Advanced Studies and Systems of Recife（里贝拉先进研究与系统中心）；Faculty of Technical Sciences University of Novi Sad（诺维萨德大学技术科学学院）
ES-Parkour: Advanced Robot Parkour with Bio-inspired Event Camera and Spiking Neural Network
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing Innovation Center of Humanoid Robotics Co., Ltd.（北京人形机器人创新中心有限公司）；Zhejiang University（浙江大学）
RSC-SNN: Exploring the Trade-off Between Adversarial Robustness and Accuracy in Spiking Neural Networks via Randomized Smoothing Coding
Chongqing University（重庆大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Xi’an Jiaotong University（西安交通大学）；Software Security Technology Company Ltd（软件安全技术有限公司）
Spiking Neural Networks Need High Frequency Information
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Brain Mind Innovation INC（脑科学与创新有限公司）；JD Explore Academy（京东探索研究院）；Northwestern University（西北大学）
Intrinsically Stable Spiking Neural Networks: Overcoming the Performance Barrier in the Absence of Batch Normalization
University of Electronic Science and Technology of China (UESTC)（电子科技大学）；Beijing Institute of Remote-Sensing Equipment（北京遥感设备研究所）；Shenzhen Institute for Advanced Study, UESTC（电子科技大学深圳高等研究院）
Current Injection Spiking Neural Network for Infrared and Visible Image Fusion
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；The Hong Kong Polytechnic University（香港理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机科学学院多媒体信息处理国家重点实验室）
Beijing University of Posts and Telecommunications（北京邮电大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Princeton University（普林斯顿大学）；The University of Edinburgh（爱丁堡大学）；Virginia Tech（弗吉尼亚理工大学）；Huawei France R & D（华为法国研发中心）
Next-generation wireless networks must support ultra-reliable, low-latency communication and intelligently manage a massive number of Internet of Things (IoT) devices in real-time, within a highly dynamic environment. This need for stringent communication quality-of-service (QoS) requirements as well as mobile edge and core intelligence can only be realized by integrating fundamental notions of artificial intelligence (AI) and machine learning across the wireless infrastructure and end-user devices. In this context, this paper provides a comprehensive tutorial that introduces the main concepts…
