Neuromorphic vision sensing (NVS)\ devices represent visual information as sequences of asynchronous discrete events (a.k.a., ``spikes'') in response to changes in scene reflectance. Unlike conventional active pixel sensing (APS), NVS allows for significantly higher event sampling rates at substantially increased energy efficiency and robustness to illumination changes. However, object classification with NVS streams cannot leverage on state-of-the-art convolutional neural networks (CNNs), since NVS does not produce frame representations. To circumvent this mismatch between sensing and processing with CNNs, we propose a compact graph representation for NVS. We couple this with novel residual graph CNN architectures and show that, when trained on spatio-temporal NVS data for object classification, such residual graph CNNs preserve the spatial and temporal coherence of spike events, while requiring less computation and memory. Finally, to address the absence of large real-world NVS datasets for complex recognition tasks, we present and make available a 100k dataset of NVS recordings of the American sign language letters, acquired with an iniLabs DAVIS240c device under real-world conditions.
NeuroLiDAR: Adaptive Frame Rate Depth Sensing via Neuromorphic Event-LiDAR Fusion
Singapore Management University（新加坡国立管理学院）；Singapore-MIT Alliance for Research and Technology Centre（新加坡-麻省理工联合研究和技术中心）；University of Technology Sydney（悉尼技术大学）
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Tsinghua University（清华大学）
Neuromorphic data, recording frameless spike events, have attracted considerable attention for the spatiotemporal information components and the event-driven processing fashion. Spiking neural networks (SNNs) represent a family of event-driven models with spatiotemporal dynamics for neuromorphic computing, which are widely benchmarked on neuromorphic data. Interestingly, researchers in the machine learning community can argue that recurrent (artificial) neural networks (RNNs) also have the capability to extract spatiotemporal features although they are not event-driven. Thus, the question of "…
Modelling and Simulation of Neuromorphic Datasets for Anomaly Detection in Computer Vision
Bio-Inspired Systems and Technologies Lab, University of York（约克大学生物启发系统与技术实验室）；Faculty of Natural Sciences, University of Stirling（斯特林大学自然科学学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；School of Electrical and Electronic Engineering, University of Manchester（曼彻斯特大学电子与电气工程学院）；Electronics and Computer Science, University of Southampton（南安普顿大学电子与计算机科学）
Energy-Aware Spike Budgeting for Continual Learning in Spiking Neural Networks for Neuromorphic Vision
Dept. of Electrical and Electronics Engineering, University of Liberal Arts Bangladesh（乌拉尔大学 bangladesh 电子与电气工程系）；Dept. of Electrical Engineering, Pennsylvania State University（宾夕法尼亚州立大学电气工程系）
Neuromorphic Vision-based Motion Segmentation with Graph Transformer Neural Network
Propulsion and Space Research Center (PSRC) at the Technology Innovation Institute (TII)（技术创新研究所推进与空间研究中心（PSRC））；Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University of Science and Technology（哈利法科技大学自主机器人系统中心（KUCARS），哈利法科技大学）；Advanced Research and Innovation Center (ARIC), Khalifa University of Science and Technology（哈利法科技大学先进研究与创新中心（ARIC），哈利法科技大学）
Developing neuromorphic intelligence on event-based datasets with Spiking Neural Networks (SNNs) has recently attracted much research attention. However, the limited size of event-based datasets makes SNNs prone to overfitting and unstable convergence. This issue remains unexplored by previous academic works. In an effort to minimize this generalization gap, we propose Neuromorphic Data Augmentation (NDA), a family of geometric augmentations specifically designed for event-based datasets with the goal of significantly stabilizing the SNN training and reducing the generalization gap between tra…
Neuromorphic visual attention for Sign-language recognition on SpiNNaker
Faculty of Information Technology, Czech Technical University in Prague（信息技术学院，布拉格捷克技术大学）；Dept. of Cybernetics, Faculty of Electrical Engineering, Czech Technical University in Prague（电子工程系，捷克技术大学）；Université de Lille, CNRS, Centrale Lille, UMR 9189 CRIStAL（里尔大学，法国国家科学研究中心，里尔中央大学，UMR 9189 CRIStAL）
With event-driven algorithms, especially the spiking neural networks (SNNs), achieving continuous improvement in neuromorphic vision processing, a more challenging event-stream-dataset is urgently needed. However, it is well known that creating an ES-dataset is a time-consuming and costly task with neuromorphic cameras like dynamic vision sensors (DVS). In this work, we propose a fast and effective algorithm termed Omnidirectional Discrete Gradient (ODG) to convert the popular computer vision dataset ILSVRC2012 into its event-stream (ES) version, generating about 1,300,000 frame-based images i…
Khalifa University of Science and Technology（哈利法科技大学）
Neuromorphic vision is a bio-inspired technology that has triggered a paradigm shift in the computer-vision community and is serving as a key-enabler for a multitude of applications. This technology has offered significant advantages including reduced power consumption, reduced processing needs, and communication speed-ups. However, neuromorphic cameras suffer from significant amounts of measurement noise. This noise deteriorates the performance of neuromorphic event-based perception and navigation algorithms. In this paper, we propose a novel noise filtration algorithm to eliminate events whi…
Tactile sensing is an essential modality for smart robots as it enables them to interact flexibly with physical objects in their environment. Recent advancements in electronic skins have led to the development of data-driven machine learning methods that exploit this important sensory modality. However, current datasets used to train such algorithms are limited to standard synchronous tactile sensors. There is a dearth of neuromorphic event-based tactile datasets, principally due to the scarcity of large-scale event-based tactile sensors. Having such datasets is crucial for the development and…
Istituto Italiano di Tecnologia（意大利理工学院）；Politecnico di Torino（都灵理工大学）；University of Groningen（格罗宁根大学）；Silicon Austria Labs（奥地利硅实验室）；Johannes Kepler University Linz（约翰内斯·开普勒林茨大学）；Western Sydney University（西悉尼大学）；University of Hertfordshire（赫特福德大学）；Friedrich Miescher Institute for Biomedical Research（弗里德里希·米歇尔生物医学研究所）；University of Basel（巴塞尔大学）
Spatio-temporal pattern recognition is a fundamental ability of the brain which is required for numerous real-world activities. Recent deep learning approaches have reached outstanding accuracies in such tasks, but their implementation on conventional embedded solutions is still very computationally and energy expensive. Tactile sensing in robotic applications is a representative example where real-time processing and energy efficiency are required. Following a brain-inspired computing approach, we propose a new benchmark for spatio-temporal tactile pattern recognition at the edge through Brai…
Real-time tracking is an important problem in computer vision in which most methods are based on the conventional cameras. Neuromorphic vision is a concept defined by incorporating neuromorphic vision sensors such as silicon retinas in vision processing system. With the development of the silicon technology, asynchronous event-based silicon retinas that mimic neuro-biological architectures has been developed in recent years. In this work, we combine the vision tracking algorithm of computer vision with the information encoding mechanism of event-based sensors which is inspired from the neural …
Optical Linear Systems Framework for Event Sensing and Computational Neuromorphic Imaging
International Centre for Neuromorphic Systems, The MARCS Institute, Western Sydney University（西悉尼大学MARCS研究所国际神经形态系统中心）；Centre for Research in Mathematics and Data Science, Western Sydney University（西悉尼大学数学与数据科学研究中心）
JingDong Group（京东集团）；Tongji University（同济大学）；Technische Universität München（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）
Robotic grasping plays an important role in the field of robotics. The current state-of-the-art robotic grasping detection systems are usually built on the conventional vision, such as RGB-D camera. Compared to traditional frame-based computer vision, neuromorphic vision is a small and young community of research. Currently, there are limited event-based datasets due to the troublesome annotation of the asynchronous event stream. Annotating large scale vision dataset often takes lots of computation resources, especially the troublesome data for video-level annotation. In this work, we consider…
Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）
In this paper, we present EBBIOT-a novel paradigm for object tracking using stationary neuromorphic vision sensors in low-power sensor nodes for the Internet of Video Things (IoVT). Different from fully event based tracking or fully frame based approaches, we propose a mixed approach where we create event-based binary images (EBBI) that can use memory efficient noise filtering algorithms. We exploit the motion triggering aspect of neuromorphic sensors to generate region proposals based on event density counts with >1000X less memory and computes compared to frame based approaches. We also prop…
NERVE: A Neuromorphic Vision and Radar Ensemble for Multi-Sensor Fusion Research
1University of Twente, Netherlands 2University of Modena；3University of Strasbourg, France 4IMEC the Netherlands, Netherlands；5Innatera, Netherlands 6Maastricht University, Netherlands
Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
A neuromorphic camera is an image sensor that emulates the human eyes capturing only changes in local brightness levels. They are widely known as event cameras, silicon retinas or dynamic vision sensors (DVS). DVS records asynchronous per-pixel brightness changes, resulting in a stream of events that encode the brightness change's time, location, and polarity. DVS consumes little power and can capture a wider dynamic range with no motion blur and higher temporal resolution than conventional frame-based cameras. Although this method of event capture results in a lower bit rate than traditional …
Khalifa University（哈利法大学）；Khalifa University of Science and Technology（哈利法科技大学）
Space exploration has witnessed revolutionary changes upon landing of the Perseverance Rover on the Martian surface and demonstrating the first flight beyond Earth by the Mars helicopter, Ingenuity. During their mission on Mars, Perseverance Rover and Ingenuity collaboratively explore the Martian surface, where Ingenuity scouts terrain information for rover's safe traversability. Hence, determining the relative poses between both the platforms is of paramount importance for the success of this mission. Driven by this necessity, this work proposes a robust relative localization system based on …
University of Western Australia（西澳大学）；University of Michigan（密歇根大学）
Dynamic vision sensors are able to operate at high temporal resolutions within resource constrained environments, though at the expense of capturing static content. The sparse nature of event streams enables efficient downstream processing tasks as they are suited for power-efficient spiking neural networks. One of the challenges associated with neuromorphic vision is the lack of interpretability of event streams. While most application use-cases do not intend for the event stream to be visually interpreted by anything other than a classification network, there is a lost opportunity to integra…
Department of Electronic & Electrical Engineering（电子与电气工程系）；University College London（伦敦大学学院）
Neuromorphic vision sensing (NVS)\ devices represent visual information as sequences of asynchronous discrete events (a.k.a., "spikes") in response to changes in scene reflectance. Unlike conventional active pixel sensing (APS), NVS allows for significantly higher event sampling rates at substantially increased energy efficiency and robustness to illumination changes. However, feature representation for NVS is far behind its APS-based counterparts, resulting in lower performance in high-level computer vision tasks. To fully utilize its sparse and asynchronous nature, we propose a compact graph…
Institute of Neuroinformatics, University of Zurich and ETH Zurich（苏黎世大学与苏黎世联邦理工学院神经信息学研究所）
The stereo-matching problem, i.e., matching corresponding features in two different views to reconstruct depth, is efficiently solved in biology. Yet, it remains the computational bottleneck for classical machine vision approaches. By exploiting the properties of event cameras, recently proposed Spiking Neural Network (SNN) architectures for stereo vision have the potential of simplifying the stereo-matching problem. Several solutions that combine event cameras with spike-based neuromorphic processors already exist. However, they are either simulated on digital hardware or tested on simplified…
University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）；Intel Labs（英特尔实验室）
Due to the high activation sparsity and use of accumulates (AC) instead of expensive multiply-and-accumulates (MAC), neuromorphic spiking neural networks (SNNs) have emerged as a promising low-power alternative to traditional DNNs for several computer vision (CV) applications. However, most existing SNNs require multiple time steps for acceptable inference accuracy, hindering real-time deployment and increasing spiking activity and, consequently, energy consumption. Recent works proposed direct encoding that directly feeds the analog pixel values in the first layer of the SNN in order to signi…
The increasing rise in machine learning and deep learning applications is requiring ever more computational resources to successfully meet the growing demands of an always-connected, automated world. Neuromorphic technologies based on Spiking Neural Network algorithms hold the promise to implement advanced artificial intelligence using a fraction of the computations and power requirements by modeling the functioning, and spiking, of the human brain. With the proliferation of tools and platforms aiding data scientists and machine learning engineers to develop the latest innovations in artificia…
Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions due to the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present the first fully neuromorphic vision-to-control pipeline for contro…
Birla Institute of Technology & Science (BITS) Pilani（比拉理工学院（BITS）皮拉尼分校）
Neuromorphic engineering is essentially the development of artificial systems, such as electronic analog circuits that employ information representations found in biological nervous systems. Despite being faster and more accurate than the human brain, computers lag behind in recognition capability. However, it is envisioned that the advancement in neuromorphics, pertaining to the fields of computer vision and image processing will provide a considerable improvement in the way computers can interpret and analyze information. In this paper, we explore the implementation of visual tasks such as i…
University of Galway（戈尔韦大学）；Xperi Corporation（Xperi公司）
Neuromorphic vision or event vision is an advanced vision technology, where in contrast to the visible camera that outputs pixels, the event vision generates neuromorphic events every time there is a brightness change which exceeds a specific threshold in the field of view (FOV). This study focuses on leveraging neuromorphic event data for roadside object detection. This is a proof of concept towards building artificial intelligence (AI) based pipelines which can be used for forward perception systems for advanced vehicular applications. The focus is on building efficient state-of-the-art obje…
Neuromorphic vision-based sensors are gaining popularity in recent years with their ability to capture Spatio-temporal events with low power sensing. These sensors record events or spikes over traditional cameras which helps in preserving the privacy of the subject being recorded. These events are captured as per-pixel brightness changes and the output data stream is encoded with time, location, and pixel intensity change information. This paper proposes and benchmarks the performance of fine-tuned conventional vision models on neuromorphic human action recognition and fall detection datasets.…
School of Electrical Engineering, Guangxi University（广西大学电气工程学院）；Baise Artificial Intelligence Innovation and Development Center（百色人工智能创新与发展中心）；School of Physical Science and Technology, Northwestern Polytechnical University（西北工业大学物理科学与技术学院）
EventZoom: A Progressive Approach to Event-Based Data Augmentation for Enhanced Neuromorphic Vision
School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Huazhong University of Science and Technology（华中科技大学）
Neuromorphic vision sensors (event cameras) are inherently suitable for spiking neural networks (SNNs) and provide novel neuromorphic vision data for this biomimetic model. Due to the spatiotemporal characteristics, novel data augmentations are required to process the unconventional visual signals of these cameras. In this paper, we propose a novel Event SpatioTemporal Fragments (ESTF) augmentation method. It preserves the continuity of neuromorphic data by drifting or inverting fragments of the spatiotemporal event stream to simulate the disturbance of brightness variations, leading to more r…
Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）
Applications in the Internet of Video Things (IoVT) domain have very tight constraints with respect to power and area. While neuromorphic vision sensors (NVS) may offer advantages over traditional imagers in this domain, the existing NVS systems either do not meet the power constraints or have not demonstrated end-to-end system performance. To address this, we improve on a recently proposed hybrid event-frame approach by using morphological image processing algorithms for region proposal and address the low-power requirement for object detection and classification by exploring various convolut…
IBM Research（IBM研究院）；SynSense（SynSense（芯仑科技））；ETH Zürich（苏黎世联邦理工学院）；University of Zürich（苏黎世大学）
Event-based dynamic vision sensors provide very sparse output in the form of spikes, which makes them suitable for low-power applications. Convolutional spiking neural networks model such event-based data and develop their full energy-saving potential when deployed on asynchronous neuromorphic hardware. Event-based vision being a nascent field, the sensitivity of spiking neural networks to potentially malicious adversarial attacks has received little attention so far. We show how white-box adversarial attack algorithms can be adapted to the discrete and sparse nature of event-based visual data…
The University of Hong Kong（香港大学）；Centre for Transformative Garment Production（变革性服装生产中心）；Wuhan University（武汉大学）；Beihang University（北京航空航天大学）；City University of Hong Kong（香港城市大学）；Texas A&M University（德克萨斯农工大学）
With the advent of neuromorphic vision sensors such as event-based cameras, a paradigm shift is required for most computer vision algorithms. Among these algorithms, optical flow estimation is a prime candidate for this process considering that it is linked to a neuromorphic vision approach. Usage of optical flow is widespread in robotics applications due to its richness and accuracy. We present a Principal Component Analysis (PCA) approach to the problem of event-based optical flow estimation. In this approach, we examine different regularization methods which efficiently enhance the estimati…
In the last decade, special purpose computing systems, such as Neuromorphic computing, have become very popular in the field of computer vision and machine learning for classification tasks. In 2015, IBM's released the TrueNorth Neuromorphic system, kick-starting a new era of Neuromorphic computing. Alternatively, Deep Learning approaches such as Deep Convolutional Neural Networks (DCNN) show almost human-level accuracies for detection and classification tasks. IBM's 2016 release of a deep learning framework for DCNNs, called Energy Efficient Deep Neuromorphic Networks (Eedn). Eedn shows promi…
A neuromorphic model of the insect visual system for natural image processing
School of Natural Sciences, Macquarie University（麦考瑞大学自然科学学院）；QUT Centre for Robotics, Queensland University of Technology（昆士兰科技大学机器人中心）；College of Medicine and Public Health, Flinders University（弗林德斯大学医学院与公共卫生学院）
Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR）资讯通信研究院）
In the field of artificial intelligence, neuromorphic computing has been around for several decades. Deep learning has however made much recent progress such that it consistently outperforms neuromorphic learning algorithms in classification tasks in terms of accuracy. Specifically in the field of image classification, neuromorphic computing has been traditionally using either the temporal or rate code for encoding static images in datasets into spike trains. It is only till recently, that neuromorphic vision sensors are widely used by the neuromorphic research community, and provides an alter…
Demonstrating the Suitability of Neuromorphic, Event-Based, Dynamic Vision Sensors for In Process Monitoring of Metallic Additive Manufacturing and Welding
See Silhouettes in Motion with Neuromorphic Vision
School of Electrical Engineering, Guangxi University（广西大学电气工程学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；SHU General Intelligent Robotics Research Institute（SHU通用智能机器人研究院）；School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications（北京邮电大学计算机科学学院（国家试点软件学院））；School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）
