Using hand gestures to answer a call or to control the radio while driving a car, is nowadays an established feature in more expensive cars. High resolution time-of-flight cameras and powerful embedded processors usually form the heart of these gesture recognition systems. This however comes with a price tag. We therefore investigate the possibility to design an algorithm that predicts hand gestures using a cheap low-resolution thermal camera with only 32x24 pixels, which is light-weight enough to run on a low-cost processor. We recorded a new dataset of over 1300 video clips for training and evaluation and propose a light-weight low-latency prediction algorithm. Our best model achieves 95.9% classification accuracy and 83% mAP detection accuracy while its processing pipeline has a latency of only one frame.
University of Verona（维罗纳大学）；Università di Bologna（博洛尼亚大学）；Università di Modena e Reggio Emilia（摩德纳雷焦艾米利亚大学）；University of Science, Ho Chi Minh City（胡志明市科学大学）；Zuse Institute Berlin（柏林祖斯研究所）
This paper presents the outcomes of a contest organized to evaluate methods for the online recognition of heterogeneous gestures from sequences of 3D hand poses. The task is the detection of gestures belonging to a dictionary of 16 classes characterized by different pose and motion features. The dataset features continuous sequences of hand tracking data where the gestures are interleaved with non-significant motions. The data have been captured using the Hololens 2 finger tracking system in a realistic use-case of mixed reality interaction. The evaluation is based not only on the detection pe…
Generalizing WiFi Gesture Recognition via Large-Model-Aware Semantic Distillation and Alignment
Institute of Advanced Technology, University of Science and Technology of China（科学技术大学先进技术研究院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）；School of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程学院）
WaveFormer: A Lightweight Transformer Model for sEMG-based Gesture Recognition
Integrated Systems Laboratory, ETH Zurich（苏黎世联邦理工学院集成系统实验室）；DEI, University of Bologna（博洛尼亚大学工程学院）；DIEF, University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学经济工程学院）
本文提出轻量级 Transformer 架构 WaveFormer，利用可学习小波变换和 WaveletConv 融合 sEMG 时频特征，以更少参数实现高准确率与 CPU 端低延迟实时手势识别。
Université de Sherbrooke（舍布鲁克大学）；Defence Research and Development Canada（加拿大国防研究与发展部）
We propose a novel low-complexity lidar gesture recognition system for mobile robot control robust to gesture variation. Our system uses a modular approach, consisting of a pose estimation module and a gesture classifier. Pose estimates are predicted from lidar scans using a Convolutional Neural Network trained using an existing stereo-based pose estimation system. Gesture classification is accomplished using a Long Short-Term Memory network and uses a sequence of estimated body poses as input to predict a gesture. Breaking down the pipeline into two modules reduces the dimensionality of the i…
Department of Electrical and Computer Engineering, University of Arizona（电子与计算机工程系，亚利桑那大学）；Department of Computing and Data Science, University of California, Berkeley（计算与数据科学系，加州大学伯克利分校）；Quandary Peak Research（Quandary Peak研究公司）；Department of Systems and Industrial Engineering, University of Arizona（系统与工业工程系，亚利桑那大学）；College of Information Sciences, University of Arizona（信息科学学院，亚利桑那大学）
Hand gestures form an intuitive means of interaction in Mixed Reality (MR) applications. However, accurate gesture recognition can be achieved only through state-of-the-art deep learning models or with the use of expensive sensors. Despite the robustness of these deep learning models, they are generally computationally expensive and obtaining real-time performance on-device is still a challenge. To this end, we propose a novel lightweight hand gesture recognition framework that works in First Person View for wearable devices. The models are trained on a GPU machine and ported on an Android sma…
Xidian University（西安电子科技大学）；Guangzhou Institute of Technology, Xidian University（西安电子科技大学广州研究院）
Spiking neural networks (SNNs), as one of the brain-inspired models, has spatio-temporal information processing capability, low power feature, and high biological plausibility. The effective spatio-temporal feature makes it suitable for event streams classification. However, neuromorphic datasets, such as N-MNIST, CIFAR10-DVS, DVS128-gesture, need to aggregate individual events into frames with a new higher temporal resolution for event stream classification, which causes high training and inference latency. In this work, we proposed a spatio-temporal compression method to aggregate individual…
Department of Computer and Electrical Engineering, Rutgers University（计算机与电气工程系，罗格斯大学）；Cloud to Street（云街（Cloud to Street））
Accurate flood detection in near real time via high resolution, high latency satellite imagery is essential to prevent loss of lives by providing quick and actionable information. Instruments and sensors useful for flood detection are only available in low resolution, low latency satellites with region re-visit periods of up to 16 days, making flood alerting systems that use such satellites unreliable. This work presents H2O-Network, a self supervised deep learning method to segment floods from satellites and aerial imagery by bridging domain gap between low and high latency satellite and coar…
In this paper, we develop four spiking neural network (SNN) models for two static American Sign Language (ASL) hand gesture classification tasks, i.e., the ASL Alphabet and ASL Digits. The SNN models are deployed on Intel's neuromorphic platform, Loihi, and then compared against equivalent deep neural network (DNN) models deployed on an edge computing device, the Intel Neural Compute Stick 2 (NCS2). We perform a comprehensive comparison between the two systems in terms of accuracy, latency, power consumption, and energy. The best DNN model achieves an accuracy of 99.93% on the ASL Alphabet dat…
Live Demonstration: Neuromorphic Radar for Gesture Recognition
Department of Electronic Systems Engineering, Indian Institute of Science, Bangalore, India 560012（电子系统工程系，印度科学院，班加罗尔，印度560012）；Department of Electrical Engineering, Indian Institute of Science, Bangalore, India 560012（电气工程系，印度科学院，班加罗尔，印度560012）
National University of Singapore（新加坡国立大学）；Singapore Institute for Neurotechnology（新加坡神经技术研究所）
In this paper, we address the challenging problem of action recognition, using event-based cameras. To recognise most gestural actions, often higher temporal precision is required for sampling visual information. Actions are defined by motion, and therefore, when using event-based cameras it is often unnecessary to re-sample the entire scene. Neuromorphic, event-based cameras have presented an alternative to visual information acquisition by asynchronously time-encoding pixel intensity changes, through temporally precise spikes (10 micro-second resolution), making them well equipped for action…
A Generative AI Integrated Multimodal Framework for Low-Latency Multi-Camera Person Re-Identification
University of Moratuwa（莫拉图瓦大学）；Zone24x7 (Pvt) Ltd（Zone24x7（私人）有限公司）；Chulalongkorn University（朱拉隆功大学）；University of Colombo（科伦坡大学）；La Trobe University（拉筹伯大学）
Renault（雷诺）；LEAT / CNRS UMR 7248（LEAT / 法国国家科学研究中心 UMR 7248）；University Côte d’Azur（蔚蓝海岸大学）
Convolutional neural networks (CNNs) are now the de facto solution for computer vision problems thanks to their impressive results and ease of learning. These networks are composed of layers of connected units called artificial neurons, loosely modeling the neurons in a biological brain. However, their implementation on conventional hardware (CPU/GPU) results in high power consumption, making their integration on embedded systems difficult. In a car for example, embedded algorithms have very high constraints in term of energy, latency and accuracy. To design more efficient computer vision algo…
