Gesture recognition is one of the most intuitive ways of interaction and has gathered particular attention for human computer interaction. Radar sensors possess multiple intrinsic properties, such as their ability to work in low illumination, harsh weather conditions, and being low-cost and compact, making them highly preferable for a gesture recognition solution. However, most literature work focuses on solutions with a limited range that is lower than a meter. We propose a novel architecture for a long-range (1m - 2m) gesture recognition solution that leverages a point cloud-based cross-learning approach from camera point cloud to 60-GHz FMCW radar point cloud, which allows learning better representations while suppressing noise. We use a variant of Dynamic Graph CNN (DGCNN) for the cross-learning, enabling us to model relationships between the points at a local and global level and to model the temporal dynamics a Bi-LSTM network is employed. In the experimental results section, we demonstrate our model's overall accuracy of 98.4% for five gestures and its generalization capability.
Aalto University（阿尔托大学）；University of Bergen（卑尔根大学）；Aalborg University（奥尔堡大学）
We present Tesla-Rapture, a gesture recognition interface for point clouds generated by mmWave Radars. State of the art gesture recognition models are either too resource consuming or not sufficiently accurate for integration into real-life scenarios using wearable or constrained equipment such as IoT devices (e.g. Raspberry PI), XR hardware (e.g. HoloLens), or smart-phones. To tackle this issue, we developed Tesla, a Message Passing Neural Network (MPNN) graph convolution approach for mmWave radar point clouds. The model outperforms the state of the art on two datasets in terms of accuracy wh…
In modern on-driving computing environments, many sensors are used for context-aware applications. This paper utilizes two deep learning models, U-Net and EfficientNet, which consist of a convolutional neural network (CNN), to detect hand gestures and remove noise in the Range Doppler Map image that was measured through a millimeter-wave (mmWave) radar. To improve the performance of classification, accurate pre-processing algorithms are essential. Therefore, a novel pre-processing approach to denoise images before entering the first deep learning model stage increases the accuracy of classific…
Robust Multimodal Learning Framework For Intake Gesture Detection Using Contactless Radar and Wearable IMU Sensors
e-Media Research Lab（e-Media研究实验室）；ESAT-STADIUS Division, KU Leuven（ESAT-STADIUS部门，KU莱顿大学）；M-Group, DistriNet, Department of Computer Science, KU Leuven（M组、DistriNet、计算机科学系，KU莱顿大学）
Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention
KU Leuven（鲁汶大学）；Vellore Institute of Technology（韦洛尔理工学院）；IMEC（比利时微电子研究中心）；Wageningen University and Research（瓦赫宁根大学及研究中心）；OnePlanet Research Center（壹行星研究中心）
The paper proposes a novel feature extraction approach for FMCW radar systems in the field of short-range gesture sensing. A light-weight processing is proposed which reduces a series of 3D radar data cubes to four 1D time signals containing information about range, azimuth angle, elevation angle and magnitude. The processing is entirely performed in the time domain without using any Fourier transformation and enables the training of a deep neural network directly on the raw time domain data. It is shown experimentally on real world data, that the proposed processing retains the same expressiv…
University of Moratuwa（莫拉图瓦大学）；Northeastern University（东北大学）；Florida International University（佛罗里达国际大学）
Transformers combined with convolutional encoders have been recently used for hand gesture recognition (HGR) using micro-Doppler signatures. We propose a vision-transformer-based architecture for HGR with multi-antenna continuous-wave Doppler radar receivers. The proposed architecture consists of three modules: a convolutional encoderdecoder, an attention module with three transformer layers, and a multi-layer perceptron. The novel convolutional decoder helps to feed patches with larger sizes to the attention module for improved feature extraction. Experimental results obtained with a dataset …
Millimeter wave (mmWave) based gesture recognition technology provides a good human computer interaction (HCI) experience. Prior works focus on the close-range gesture recognition, but fall short in range extension, i.e., they are unable to recognize gestures more than one meter away from considerable noise motions. In this paper, we design a long-range gesture recognition model which utilizes a novel data processing method and a customized artificial Convolutional Neural Network (CNN). Firstly, we break down gestures into multiple reflection points and extract their spatial-temporal features …
FMCW radar could detect object's range, speed and Angleof-Arrival, advantages are robust to bad weather, good range resolution, and good speed resolution. In this paper, we consider the FMCW radar as a novel interacting interface on laptop. We merge sequences of object's range, speed, azimuth information into single input, then feed to a convolution neural network to learn spatial and temporal patterns. Our model achieved 96% accuracy on test set and real-time test.
We introduce a simple but effective technique in automatic hand gesture recognition using radar. The proposed technique classifies hand gestures based on the envelopes of their micro-Doppler signatures. These envelopes capture the distinctions among different hand movements and their corresponding positive and negative Doppler frequencies which are generated during each gesture act. We detect the positive and negative envelopes separately, and form a feature vector of their augmentation. We use the $k$-nearest neighbor ($k$NN) classifier and Manhattan distance (L1) measure, in lieu of Euclidea…
ETH Zürich（苏黎世联邦理工学院）；University of Bologna（博洛尼亚大学）
This work proposes a low-power high-accuracy embedded hand-gesture recognition algorithm targeting battery-operated wearable devices using low power short-range RADAR sensors. A 2D Convolutional Neural Network (CNN) using range frequency Doppler features is combined with a Temporal Convolutional Neural Network (TCN) for time sequence prediction. The final algorithm has a model size of only 46 thousand parameters, yielding a memory footprint of only 92 KB. Two datasets containing 11 challenging hand gestures performed by 26 different people have been recorded containing a total of 20,210 gestur…
Radar processing via spiking neural networks (SNNs) has recently emerged as a solution in the field of ultra-low-power wireless human-computer interaction. Compared to traditional energy- and area-hungry deep learning methods, SNNs are significantly more energy efficient and can be deployed in the growing number of compact SNN accelerator chips, making them a better solution for ubiquitous IoT applications. We propose a novel SNN strategy for radar gesture recognition, achieving more than 91% of accuracy on two different radar datasets. Our work significantly differs from previous approaches a…
Interpretable Rule-Based System for Radar-Based Gesture Sensing: Enhancing Transparency and Personalization in AI
Technical University Munich（慕尼黑工业大学）；Infineon Technologies AG（英飞凌科技股份公司）；Johannes Kepler University Linz（林茨约翰内斯开普勒大学）；Software Competence Center Hagenberg GmbH (SCCH)（哈根堡软件能力中心有限公司）
In this paper, a real-time signal processing frame-work based on a 60 GHz frequency-modulated continuous wave (FMCW) radar system to recognize gestures is proposed. In order to improve the robustness of the radar-based gesture recognition system, the proposed framework extracts a comprehensive hand profile, including range, Doppler, azimuth and elevation, over multiple measurement-cycles and encodes them into a feature cube. Rather than feeding the range-Doppler spectrum sequence into a deep convolutional neural network (CNN) connected with recurrent neural networks, the proposed framework tak…
A recent paper (Neural Networks, {\bf 132} (2020), 253-268) introduces a straightforward and simple kernel based approximation for manifold learning that does not require the knowledge of anything about the manifold, except for its dimension. In this paper, we examine how the pointwise error in approximation using least squares optimization based on similarly localized kernels depends upon the data characteristics and deteriorates as one goes away from the training data. The theory is presented with an abstract localized kernel, which can utilize any prior knowledge about the data being locate…
We present an approach to automatically generate semantic labels for real recordings of automotive range-Doppler (RD) radar spectra. Such labels are required when training a neural network for object recognition from radar data. The automatic labeling approach rests on the simultaneous recording of camera and lidar data in addition to the radar spectrum. By warping radar spectra into the camera image, state-of-the-art object recognition algorithms can be applied to label relevant objects, such as cars, in the camera image. The warping operation is designed to be fully differentiable, which all…
In this paper, we demonstrate the ability to recognize hand gestures in a non-contact, wireless fashion using only incoherent light signals reflected from a human subject. Fundamentally distinguished from radar, lidar and camera-based sensing systems, this sensing modality uses only a low-cost light source (e.g., LED) and sensor (e.g., photodetector). The light-wave-based gesture recognition system identifies different gestures from the variations in light intensity reflected from the subject's hand within a short (20-35 cm) range. As users perform different gestures, scattered light forms uni…
Resonate-and-Fire Spiking Neurons for Target Detection and Hand Gesture Recognition: A Hybrid Approach
University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Infineon Technologies AG（英飞凌科技股份公司）；Institute for Electronics Engineering（电子工程研究所）；Otto-von-Guericke-University Magdeburg（马格德堡奥托·冯·格里克大学）
University of Washington（华盛顿大学）；Silkwave Holdings Limited（Silkwave控股有限公司）
Radar is usually more robust than the camera in severe driving scenarios, e.g., weak/strong lighting and bad weather. However, unlike RGB images captured by a camera, the semantic information from the radar signals is noticeably difficult to extract. In this paper, we propose a deep radar object detection network (RODNet), to effectively detect objects purely from the carefully processed radar frequency data in the format of range-azimuth frequency heatmaps (RAMaps). Three different 3D autoencoder based architectures are introduced to predict object confidence distribution from each snippet of…
University of Oxford（牛津大学）；University of Edinburgh（爱丁堡大学）
mmWave FMCW radar has attracted huge amount of research interest for human-centered applications in recent years, such as human gesture/activity recognition. Most existing pipelines are built upon conventional Discrete Fourier Transform (DFT) pre-processing and deep neural network classifier hybrid methods, with a majority of previous works focusing on designing the downstream classifier to improve overall accuracy. In this work, we take a step back and look at the pre-processing module. To avoid the drawbacks of conventional DFT pre-processing, we propose a learnable pre-processing module, na…
Non-verbal communication is part of our regular conversation, and multiple gestures are used to exchange information. Among those gestures, pointing is the most important one. If such gestures cannot be perceived by other team members, e.g. by blind and visually impaired people (BVIP), they lack important information and can hardly participate in a lively workflow. Thus, this paper describes a system for detecting such pointing gestures to provide input for suitable output modalities to BVIP. Our system employs an RGB-D camera to recognize the pointing gestures performed by the users. The syst…
