Tactile perception is crucial for a variety of robot tasks including grasping and in-hand manipulation. New advances in flexible, event-driven, electronic skins may soon endow robots with touch perception capabilities similar to humans. These electronic skins respond asynchronously to changes (e.g., in pressure, temperature), and can be laid out irregularly on the robot's body or end-effector. However, these unique features may render current deep learning approaches such as convolutional feature extractors unsuitable for tactile learning. In this paper, we propose a novel spiking graph neural network for event-based tactile object recognition. To make use of local connectivity of taxels, we present several methods for organizing the tactile data in a graph structure. Based on the constructed graphs, we develop a spiking graph convolutional network. The event-driven nature of spiking neural network makes it arguably more suitable for processing the event-based data. Experimental results on two tactile datasets show that the proposed method outperforms other state-of-the-art spiking methods, achieving high accuracies of approximately 90\% when classifying a variety of different household objects.
Tactile sensing is essential for a variety of daily tasks. And recent advances in event-driven tactile sensors and Spiking Neural Networks (SNNs) spur the research in related fields. However, SNN-enabled event-driven tactile learning is still in its infancy due to the limited representation abilities of existing spiking neurons and high spatio-temporal complexity in the event-driven tactile data. In this paper, to improve the representation capability of existing spiking neurons, we propose a novel neuron model called "location spiking neuron", which enables us to extract features of event-bas…
Tactile sensing is an essential modality for smart robots as it enables them to interact flexibly with physical objects in their environment. Recent advancements in electronic skins have led to the development of data-driven machine learning methods that exploit this important sensory modality. However, current datasets used to train such algorithms are limited to standard synchronous tactile sensors. There is a dearth of neuromorphic event-based tactile datasets, principally due to the scarcity of large-scale event-based tactile sensors. Having such datasets is crucial for the development and…
University of California, Berkeley（加州大学伯克利分校）；Facebook AI Research（脸书人工智能研究院）
Much of the literature on robotic perception focuses on the visual modality. Vision provides a global observation of a scene, making it broadly useful. However, in the domain of robotic manipulation, vision alone can sometimes prove inadequate: in the presence of occlusions or poor lighting, visual object identification might be difficult. The sense of touch can provide robots with an alternative mechanism for recognizing objects. In this paper, we study the problem of touch-based instance recognition. We propose a novel framing of the problem as multi-modal recognition: the goal of our system…
Contact Status Recognition and Slip Detection with a Bio-inspired Tactile Hand
School of Mechanical Engineering, Southeast University Nanjing 211189, China；School of Electronic Science \& Engineering, Southeast University Nanjing 211189, China
Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years. However, the progress in these directions is limited by the small set of objects available -- synthetic objects are not realistic enough and are mostly centered around geometry, while real object datasets such as YCB are often practically challenging and unstable to acquire due to international shipping, inventory, and financial cost. We present ObjectFolder, a dataset of 100 virtualized objects that addresses both challenges with two key innovations. First, ObjectFolder encodes t…
The human visual system contains a hierarchical sequence of modules that take part in visual perception at superordinate, basic, and subordinate categorization levels. During the last decades, various computational models have been proposed to mimic the hierarchical feed-forward processing of visual cortex, but many critical characteristics of the visual system, such actual neural processing and learning mechanisms, are ignored. Pursuing the line of biological inspiration, we propose a computational model for object recognition in different categorization levels, in which a spiking neural netw…
Dalian University of Technology（大连理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））
SGED: A Benchmark dataset for Performance Evaluation of Spiking Gesture Emotion Recognition
Shandong Massive Information Technology Research Institute（山东海量信息技术研究院）；State Key Laboratory of High-end Server & Storage Technology（高端服务器与存储技术国家重点实验室）；Inspur (Beijing) Electronic Information Industry Co., Ltd.（浪潮（北京）电子信息产业有限公司）；Inspur Electronic Information Industry Co., Ltd.（浪潮电子信息产业股份有限公司）；Intel Labs China（英特尔中国研究院）
Istituto Italiano di Tecnologia（意大利理工学院）；Politecnico di Torino（都灵理工大学）；University of Groningen（格罗宁根大学）；Silicon Austria Labs（奥地利硅实验室）；Johannes Kepler University Linz（约翰内斯·开普勒林茨大学）；Western Sydney University（西悉尼大学）；University of Hertfordshire（赫特福德大学）；Friedrich Miescher Institute for Biomedical Research（弗里德里希·米歇尔生物医学研究所）；University of Basel（巴塞尔大学）
Spatio-temporal pattern recognition is a fundamental ability of the brain which is required for numerous real-world activities. Recent deep learning approaches have reached outstanding accuracies in such tasks, but their implementation on conventional embedded solutions is still very computationally and energy expensive. Tactile sensing in robotic applications is a representative example where real-time processing and energy efficiency are required. Following a brain-inspired computing approach, we propose a new benchmark for spatio-temporal tactile pattern recognition at the edge through Brai…
SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Beijing Institute Of Technology（北京理工大学）；Peng Cheng Laboratory（鹏城实验室）；National Engineering Laboratory for Video Technology, School of Electronics Engineering and Computer Science, Peking University（视频技术国家工程实验室，北京大学电子工程与计算机科学学院）
Benchmarking Microsaccade Recognition with Event Cameras: A Novel Dataset and Evaluation
C3I Imaging Lab, School of Engineering, University of Galway, Ireland（C3I成像实验室，工程学院，Galway大学，爱尔兰）；School of Computer Science, University of Galway, Ireland（计算机科学学院，Galway大学，爱尔兰）；Department of Electronic and Electrical Engineering, Trinity College Dublin, Ireland（电子与电气工程系，Trinity College Dublin大学，爱尔兰）
Center for Advanced Systems Understanding (CASUS)（先进系统理解中心（CASUS））；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（亥姆霍兹德累斯顿罗森多夫研究中心（HZDR））；Institute of Theoretical and Applied Informatics, Polish Academy of Sciences（波兰科学院理论与应用信息学研究所）；Laboratoire I3S, University Cote d’Azur（蔚蓝海岸大学I3S实验室）；Jadavpur University（贾达普大学）
Classical Random Neural Networks (RNNs) have demonstrated effective applications in decision making, signal processing, and image recognition tasks. However, their implementation has been limited to deterministic digital systems that output probability distributions in lieu of stochastic behaviors of random spiking signals. We introduce the novel class of supervised Random Quantum Neural Networks (RQNNs) with a robust training strategy to better exploit the random nature of the spiking RNN. The proposed RQNN employs hybrid classical-quantum algorithms with superposition state and amplitude enc…
Invariant neuromorphic representations of tactile stimuli improve robustness of a real-time texture classification system
Johns Hopkins School of Medicine（约翰斯·霍普金斯医学院）；University of Chicago（芝加哥大学）；National Institutes of Health（美国国立卫生研究院）；Vanderbilt University（范德堡大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；The University of North Texas Health Science Center（北得克萨斯健康科学中心大学）；Infinite Biomedical Technologies, LLC.（无限生物医学技术有限责任公司）
To increase the quality of citizens' lives, we designed a personalized smart chair system to recognize sitting behaviors. The system can receive surface pressure data from the designed sensor and provide feedback for guiding the user towards proper sitting postures. We used a liquid state machine and a logistic regression classifier to construct a spiking neural network for classifying 15 sitting postures. To allow this system to read our pressure data into the spiking neurons, we designed an algorithm to encode map-like data into cosine-rank sparsity data. The experimental results consisting …
We study the Human Activity Recognition (HAR) task, which predicts user daily activity based on time series data from wearable sensors. Recently, researchers use end-to-end Artificial Neural Networks (ANNs) to extract the features and perform classification in HAR. However, ANNs pose a huge computation burden on wearable devices and lack temporal feature extraction. In this work, we leverage Spiking Neural Networks (SNNs)--an architecture inspired by biological neurons--to HAR tasks. SNNs allow spatio-temporal extraction of features and enjoy low-power computation with binary spikes. We conduc…
Touch is the first sense among human senses. Not only that, but it is also one of the most important senses that are indispensable. However, compared to sight and hearing, it is often neglected. In particular, since humans use the tactile sense of the skin to recognize and manipulate objects, without tactile sensation, it is very difficult to recognize or skillfully manipulate objects. In addition, the importance and interest of haptic technology related to touch are increasing with the development of technologies such as VR and AR in recent years. So far, the focus is only on haptic technolog…
Radar processing via spiking neural networks (SNNs) has recently emerged as a solution in the field of ultra-low-power wireless human-computer interaction. Compared to traditional energy- and area-hungry deep learning methods, SNNs are significantly more energy efficient and can be deployed in the growing number of compact SNN accelerator chips, making them a better solution for ubiquitous IoT applications. We propose a novel SNN strategy for radar gesture recognition, achieving more than 91% of accuracy on two different radar datasets. Our work significantly differs from previous approaches a…
Resonate-and-Fire Spiking Neurons for Target Detection and Hand Gesture Recognition: A Hybrid Approach
University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Infineon Technologies AG（英飞凌科技股份公司）；Institute for Electronics Engineering（电子工程研究所）；Otto-von-Guericke-University Magdeburg（马格德堡奥托·冯·格里克大学）
Computation on a large volume of data at high speed and low power requires energy-efficient computing architectures. Spiking neural network (SNN) with bio-inspired spike-timing-dependent plasticity learning (STDP) is a promising solution for energy-efficient neuromorphic systems than conventional artificial neural network (ANN). Previous works on SNN with STDP learning primarily uses memristive devices which are difficult to fabricate. Some reported works on SNN makes use of memristor macro models, which are software-based and cannot give complete insight into circuit implementation challenges…
Artificial Intelligence Thrust, HKUST (Guangzhou)（香港科技大学（广州）人工智能方向）；School of Computing and Information Technology, Great Bay University（湾区大学计算机与信息技术学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Department of Computer Science and Engineering, HKUST（香港科技大学计算机科学与工程系）；Guangzhou HKUST Fok Ying Tung Research Institute（广州香港科技大学霍英东研究院）
