Advancing Spiking Neural Networks towards Multiscale Spatiotemporal Interaction Learning
Liaoning Technical University（辽宁工程技术大学）；University of Electronic Science and Technology of China（电子科技大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Yimeng Shan, Malu Zhang, Rui-jie Zhu, Xuerui Qiu, Jason K. Eshraghian, Haicheng Qu
Recent advancements in neuroscience research have propelled the development of Spiking Neural Networks (SNNs), which not only have the potential to further advance neuroscience research but also serve as an energy-efficient alternative to Artificial Neural Networks (ANNs) due to their spike-driven characteristics. However, previous studies often neglected the multiscale information and its spatiotemporal correlation between event data, leading SNN models to approximate each frame of input events as static images. We hypothesize that this oversimplification significantly contributes to the performance gap between SNNs and traditional ANNs. To address this issue, we have designed a Spiking Multiscale Attention (SMA) module that captures multiscale spatiotemporal interaction information. Furthermore, we developed a regularization method named Attention ZoneOut (AZO), which utilizes spatiotemporal attention weights to reduce the model's generalization error through pseudo-ensemble training. Our approach has achieved state-of-the-art results on mainstream neural morphology datasets. Additionally, we have reached a performance of 77.1% on the Imagenet-1K dataset using a 104-layer ResNet architecture enhanced with SMA and AZO. This achievement confirms the state-of-the-art performance of SNNs with non-transformer architectures and underscores the effectiveness of our method in bridging the performance gap between SNN models and traditional ANN models.
Huazhong University of Science and Technology（华中科技大学）
Neuromorphic vision sensors (event cameras) simulate biological visual perception systems and have the advantages of high temporal resolution, less data redundancy, low power consumption, and large dynamic range. Since both events and spikes are modeled from neural signals, event cameras are inherently suitable for spiking neural networks (SNNs), which are considered promising models for artificial intelligence (AI) and theoretical neuroscience. However, the unconventional visual signals of these cameras pose a great challenge to the robustness of spiking neural networks. In this paper, we pro…
University of Oulu（奥卢大学）；École de Technologie Supérieure（高等技术学院）；VTT Technical Research Centre of Finland Ltd（芬兰VTT技术研究中心有限公司）
Video-based analysis of facial expressions has been increasingly applied to infer health states of individuals, such as depression and pain. Among the existing approaches, deep learning models composed of structures for multiscale spatiotemporal processing have shown strong potential for encoding facial dynamics. However, such models have high computational complexity, making for a difficult deployment of these solutions. To address this issue, we introduce a new technique to decompose the extraction of multiscale spatiotemporal features. Particularly, a building block structure called Decompo…
SOKENDAI (Graduate University for Advanced Studies)（综合研究大学院大学）；National Institute of Informatics（国立信息学研究所）
Detecting salient objects from a video requires exploiting both spatial and temporal knowledge included in the video. We propose a novel region-based multiscale spatiotemporal saliency detection method for videos, where static features and dynamic features computed from the low and middle levels are combined together. Our method utilizes such combined features spatially over each frame and, at the same time, temporally across frames using consistency between consecutive frames. Saliency cues in our method are analyzed through a multiscale segmentation model, and fused across scale levels, yiel…
Huazhong University of Science and Technology（华中科技大学）
Neuromorphic vision sensors (event cameras) are inherently suitable for spiking neural networks (SNNs) and provide novel neuromorphic vision data for this biomimetic model. Due to the spatiotemporal characteristics, novel data augmentations are required to process the unconventional visual signals of these cameras. In this paper, we propose a novel Event SpatioTemporal Fragments (ESTF) augmentation method. It preserves the continuity of neuromorphic data by drifting or inverting fragments of the spatiotemporal event stream to simulate the disturbance of brightness variations, leading to more r…
Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformer
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology（计算机科学与控制工程学院，深圳大学）；Department of Electrical and Computer Engineering, University of Alberta（电气与计算机工程系，阿尔伯塔大学）
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Drexel University（德雷塞尔大学）；National Research Council of Canada（加拿大国家研究委员会）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
The Locally Competitive Algorithm (LCA) uses local competition between non-spiking leaky integrator neurons to infer sparse representations, allowing for potentially real-time execution on massively parallel neuromorphic architectures such as Intel's Loihi processor. Here, we focus on the problem of inferring sparse representations from streaming video using dictionaries of spatiotemporal features optimized in an unsupervised manner for sparse reconstruction. Non-spiking LCA has previously been used to achieve unsupervised learning of spatiotemporal dictionaries composed of convolutional kerne…
This paper proposes an unsupervised address event representation (AER) object recognition approach. The proposed approach consists of a novel multiscale spatio-temporal feature (MuST) representation of input AER events and a spiking neural network (SNN) using spike-timing-dependent plasticity (STDP) for object recognition with MuST. MuST extracts the features contained in both the spatial and temporal information of AER event flow, and meanwhile forms an informative and compact feature spike representation. We show not only how MuST exploits spikes to convey information more effectively, but a…
MS-LSTM: Exploring Spatiotemporal Multiscale Representations in Video Prediction Domain
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；International Research Institute for Artificial Intelligence, Harbin Institute of Technology（哈尔滨工业大学国际人工智能研究院）
NeuroDETR: Multiscale Discrete Spatiotemporal Tokenization for Asynchronous EEG Event Detection
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Academy（中关村学院）；Wuhan University（武汉大学）
Temporal-Aware Spiking Transformer Hashing Based on 3D-DWT
Faculty of Electrical Engineering and Computer Science, Ningbo University（宁波大学电气工程与计算机科学学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
National Institute of Technology Warangal（瓦朗加尔国家理工学院）；Institute for Development and Research in Banking Technology（银行技术发展与研究院）
Video Salient Document Detection (VSDD) is an essential task of practical computer vision, which aims to highlight visually salient document regions in video frames. Previous techniques for VSDD focus on learning features without considering the cooperation among and across the appearance and motion cues and thus fail to perform in practical scenarios. Moreover, most of the previous techniques demand high computational resources, which limits the usage of such systems in resource-constrained settings. To handle these issues, we propose VS-Net, which captures multi-scale spatiotemporal informat…
Nanjing University（南京大学）；New York University（纽约大学）
Over the past two decades, traditional block-based video coding has made remarkable progress and spawned a series of well-known standards such as MPEG-4, H.264/AVC and H.265/HEVC. On the other hand, deep neural networks (DNNs) have shown their powerful capacity for visual content understanding, feature extraction and compact representation. Some previous works have explored the learnt video coding algorithms in an end-to-end manner, which show the great potential compared with traditional methods. In this paper, we propose an end-to-end deep neural video coding framework (NVC), which uses vari…
Columbia University（哥伦比亚大学）；Lamont Doherty Earth Observatory（拉蒙特-多尔蒂地球观测站）
In presenting an irrigation detection methodology that leverages multiscale satellite imagery of vegetation abundance, this paper introduces a process to supplement limited ground-collected labels and ensure classifier applicability in an area of interest. Spatiotemporal analysis of MODIS 250m Enhanced Vegetation Index (EVI) timeseries characterizes native vegetation phenologies at regional scale to provide the basis for a continuous phenology map that guides supplementary label collection over irrigated and non-irrigated agriculture. Subsequently, validated dry season greening and senescence …
Neuromorphic vision sensor is a new bio-inspired imaging paradigm that reports asynchronous, continuously per-pixel brightness changes called `events' with high temporal resolution and high dynamic range. So far, the event-based image reconstruction methods are based on artificial neural networks (ANN) or hand-crafted spatiotemporal smoothing techniques. In this paper, we first implement the image reconstruction work via fully spiking neural network (SNN) architecture. As the bio-inspired neural networks, SNNs operating with asynchronous binary spikes distributed over time, can potentially lea…
Noisy group neurons with synchronous resetting for high-performance spiking neural networks
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Center for Intersection of Mathematics and Life Sciences, Xi’an Jiaotong University（西安交通大学数学与生命科学交叉中心）；School of Life Science and Technology, Xi’an Jiaotong University（西安交通大学生命科学与技术学院）；Research Center for Brain-inspired Intelligence, Xi’an Jiaotong University（西安交通大学类脑智能研究中心）
This paper presents a novel hierarchical spatiotemporal orientation representation for spacetime image analysis. It is designed to combine the benefits of the multilayer architecture of ConvNets and a more controlled approach to spacetime analysis. A distinguishing aspect of the approach is that unlike most contemporary convolutional networks no learning is involved; rather, all design decisions are specified analytically with theoretical motivations. This approach makes it possible to understand what information is being extracted at each stage and layer of processing as well as to minimize h…
Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；Center for Long-term Artificial Intelligence（长期人工智能中心）；Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Chinese Academy of Sciences（中国科学院脑认知与脑智能技术重点实验室）
Benchmarking Microsaccade Recognition with Event Cameras: A Novel Dataset and Evaluation
C3I Imaging Lab, School of Engineering, University of Galway, Ireland（C3I成像实验室，工程学院，Galway大学，爱尔兰）；School of Computer Science, University of Galway, Ireland（计算机科学学院，Galway大学，爱尔兰）；Department of Electronic and Electrical Engineering, Trinity College Dublin, Ireland（电子与电气工程系，Trinity College Dublin大学，爱尔兰）
NeuroMoCo: A Neuromorphic Momentum Contrast Learning Method for Spiking Neural Networks
Southwest University（西南大学）；Chongqing Key Laboratory of Brain Inspired Computing and Intelligent Chips（重庆脑启发计算与智能芯片重点实验室）；University of Technology Sydney（悉尼科技大学）
SDTrack: A Baseline for Event-based Tracking via Spiking Neural Networks
University of Electronic Science and Technology of China（电子科学与技术大学）；Shenzhen Loop Area Institute（深圳河套学院）；Liaoning Technical University（辽宁技术大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
Deep Directly-Trained Spiking Neural Networks for Object Detection
University of Chinese Academy of Sciences（中国科学院大学）；Xi’an Jiaotong University（西安交通大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua University（清华大学）；Peking University（北京大学）；Huawei Technologies Co Ltd.（华为技术有限公司）
FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks
Zhejiang University-University of Illinois Urbana Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院）；The College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）
EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-based Detection with Recurrent Spiking Neural Networks
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；State Key Laboratory of Brain-Machine Intelligence, Zhejiang University（浙江大学脑机智能国家重点实验室）
