Hand Gesture Recognition from Doppler Radar Signals Using Echo State Networks
Department of Computer Science, Nagoya Institute of Technology（名古屋技术大学计算机科学系）；International Research Center for Neurointelligence, The University of Tokyo（东京大学神经智能国际研究中心）
Hand gesture recognition (HGR) is a fundamental technology in human computer interaction (HCI).In particular, HGR based on Doppler radar signals is suited for in-vehicle interfaces and robotic systems, necessitating lightweight and computationally efficient recognition techniques. However, conventional deep learning-based methods still suffer from high computational costs. To address this issue, we propose an Echo State Network (ESN) approach for radar-based HGR, using frequency-modulated-continuous-wave (FMCW) radar signals. Raw radar data is first converted into feature maps, such as range-time and Doppler-time maps, which are then fed into one or more recurrent neural network-based reservoirs. The obtained reservoir states are processed by readout classifiers, including ridge regression, support vector machines, and random forests. Comparative experiments demonstrate that our method outperforms existing approaches on an 11-class HGR task using the Soli dataset and surpasses existing deep learning models on a 4-class HGR task using the Dop-NET dataset. The results indicate that parallel processing using multi-reservoir ESNs are effective for recognizing temporal patterns from the multiple different feature maps in the time-space and time-frequency domains. Our ESN approaches achieve high recognition performance with low computational cost in HGR, showing great potential for more advanced HCI technologies, especially in resource-constrained environments.
TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Auto-Regression
Guangdong-Hongkong-Macao Greater Bay Area Weather Research Center for Monitoring Warning and Forecasting (Shenzhen Institute of Meteorological Innovation)（粤港澳大湾区气象监测预警预报中心（深圳气象创新研究院））；Australian Artificial Intelligence Institute, School of Computer Science, FEIT, University of Technology Sydney（悉尼科技大学计算机科学学院澳大利亚人工智能研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；School of Information Science and Technology, Guangdong University of Foreign Studies（广东外语外贸大学信息科学与技术学院）；Shenzhen Meteorological Bureau / Shenzhen Key laboratory of severe weather in south China（深圳市气象局 / 深圳市南方强天气重点实验室）
Robust Multimodal Learning Framework For Intake Gesture Detection Using Contactless Radar and Wearable IMU Sensors
e-Media Research Lab（e-Media研究实验室）；ESAT-STADIUS Division, KU Leuven（ESAT-STADIUS部门，KU莱顿大学）；M-Group, DistriNet, Department of Computer Science, KU Leuven（M组、DistriNet、计算机科学系，KU莱顿大学）
University of Moratuwa（莫拉图瓦大学）；Northeastern University（东北大学）；Florida International University（佛罗里达国际大学）
Transformers combined with convolutional encoders have been recently used for hand gesture recognition (HGR) using micro-Doppler signatures. We propose a vision-transformer-based architecture for HGR with multi-antenna continuous-wave Doppler radar receivers. The proposed architecture consists of three modules: a convolutional encoderdecoder, an attention module with three transformer layers, and a multi-layer perceptron. The novel convolutional decoder helps to feed patches with larger sizes to the attention module for improved feature extraction. Experimental results obtained with a dataset …
Technische Universität Dresden（德累斯顿工业大学）；Ghent University（根特大学）；Princeton University（普林斯顿大学）
Echo State Networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the $K$-Means algorithm on the training data. We show that for a large variety of datasets this initialization pe…
ETH Zürich（苏黎世联邦理工学院）；University of Bologna（博洛尼亚大学）
This work proposes a low-power high-accuracy embedded hand-gesture recognition algorithm targeting battery-operated wearable devices using low power short-range RADAR sensors. A 2D Convolutional Neural Network (CNN) using range frequency Doppler features is combined with a Temporal Convolutional Neural Network (TCN) for time sequence prediction. The final algorithm has a model size of only 46 thousand parameters, yielding a memory footprint of only 92 KB. Two datasets containing 11 challenging hand gestures performed by 26 different people have been recorded containing a total of 20,210 gestur…
When marine radar target detection meets pretrained large language models
Department of Electronic Engineering, Tsinghua University（电子工程系，清华大学）；State Key Laboratory of Space Network and Communications, Tsinghua University（空间网络与通信国家重点实验室，清华大学）；Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University（深圳 ubiquitous 数据赋能重点实验室，清华大学深圳国际研究生院）
Aalto University（阿尔托大学）；University of Bergen（卑尔根大学）；Aalborg University（奥尔堡大学）
We present Tesla-Rapture, a gesture recognition interface for point clouds generated by mmWave Radars. State of the art gesture recognition models are either too resource consuming or not sufficiently accurate for integration into real-life scenarios using wearable or constrained equipment such as IoT devices (e.g. Raspberry PI), XR hardware (e.g. HoloLens), or smart-phones. To tackle this issue, we developed Tesla, a Message Passing Neural Network (MPNN) graph convolution approach for mmWave radar point clouds. The model outperforms the state of the art on two datasets in terms of accuracy wh…
Infineon Technologies AG（英飞凌科技公司）；Friedrich-Alexander-University Erlangen-Nuremberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Technical University of Munich（慕尼黑工业大学）
Gesture recognition is one of the most intuitive ways of interaction and has gathered particular attention for human computer interaction. Radar sensors possess multiple intrinsic properties, such as their ability to work in low illumination, harsh weather conditions, and being low-cost and compact, making them highly preferable for a gesture recognition solution. However, most literature work focuses on solutions with a limited range that is lower than a meter. We propose a novel architecture for a long-range (1m - 2m) gesture recognition solution that leverages a point cloud-based cross-lear…
Learning Interpretable Rules from Neural Networks: Neurosymbolic AI for Radar Hand Gesture Recognition
Technical University of Munich（慕尼黑技术大学）；Infineon Technologies AG（英飞凌科技有限公司）；Johannes Kepler University Linz（约翰·开普勒大学林茨分校）；Software Competence Center Hagenberg GmbH (SCCH)（哈根贝格软件竞争力中心（SCCH））
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Guiyang Academy of Information Technology（贵阳信息技术研究院）；NTT DATA Corporation（NTT DATA株式会社）；NTT DATA Institute of Management Consulting Inc.（NTT DATA经营研究所）
With the goal of predicting the future rainfall intensity in a local region over a relatively short period time, precipitation nowcasting has been a long-time scientific challenge with great social and economic impact. The radar echo extrapolation approaches for precipitation nowcasting take radar echo images as input, aiming to generate future radar echo images by learning from the historical images. To effectively handle complex and high non-stationary evolution of radar echoes, we propose to decompose the movement into optical flow field motion and morphologic deformation. Following this id…
Traditional vision-based hand gesture recognition systems is limited under dark circumstances. In this paper, we build a hand gesture recognition system based on microwave transceiver and deep learning algorithm. A Doppler radar sensor with dual receiving channels at 5.8GHz is used to acquire a big database of hand gestures signals. The received hand gesture signals are then processed with time-frequency analysis. Based on these big databases of hand gesture, we propose a new machine learning architecture called deformable deep convolutional generative adversarial network. Experimental results…
A recent paper (Neural Networks, {\bf 132} (2020), 253-268) introduces a straightforward and simple kernel based approximation for manifold learning that does not require the knowledge of anything about the manifold, except for its dimension. In this paper, we examine how the pointwise error in approximation using least squares optimization based on similarly localized kernels depends upon the data characteristics and deteriorates as one goes away from the training data. The theory is presented with an abstract localized kernel, which can utilize any prior knowledge about the data being locate…
Do Echo Top Heights Improve Deep Learning Nowcasts?
Kempelen Institute of Intelligent Technologies（智能技术研究所）；Faculty of information Technology（信息技术学院）；Brno University of Technology（布拉格技术大学）；Department of Geoscience and Remote Sensing（地质与遥感系）；Delft University of Technology（代尔夫特技术大学）
QUALCOMM Wireless Communication Technologies (China) Limited（高通无线通信技术（中国）有限公司）；Qualcomm AI Research（高通人工智能研究院）；Qualcomm Technologies Netherlands B.V.（高通科技荷兰有限公司）；Qualcomm Technologies, Inc.（高通科技公司）
We explore the feasibility of AI assisted hand-gesture recognition using 802.11ad 60GHz (mmWave) technology in smartphones. Range-Doppler information (RDI) is obtained by using pulse Doppler radar for gesture recognition. We built a prototype system, where radar sensing and WLAN communication waveform can coexist by time-division duplex (TDD), to demonstrate the real-time hand-gesture inference. It can gather sensing data and predict gestures within 100 milliseconds. First, we build the pipeline for the real-time feature processing, which is robust to occasional frame drops in the data stream.…
Resonate-and-Fire Spiking Neurons for Target Detection and Hand Gesture Recognition: A Hybrid Approach
University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Infineon Technologies AG（英飞凌科技股份公司）；Institute for Electronics Engineering（电子工程研究所）；Otto-von-Guericke-University Magdeburg（马格德堡奥托·冯·格里克大学）
University of Oxford（牛津大学）；University of Edinburgh（爱丁堡大学）
mmWave FMCW radar has attracted huge amount of research interest for human-centered applications in recent years, such as human gesture/activity recognition. Most existing pipelines are built upon conventional Discrete Fourier Transform (DFT) pre-processing and deep neural network classifier hybrid methods, with a majority of previous works focusing on designing the downstream classifier to improve overall accuracy. In this work, we take a step back and look at the pre-processing module. To avoid the drawbacks of conventional DFT pre-processing, we propose a learnable pre-processing module, na…
Given their substantial success in addressing a wide range of computer vision challenges, Convolutional Neural Networks (CNNs) are increasingly being used in smart home applications, with many of these applications relying on the automatic recognition of human activities. In this context, low-power radar devices have recently gained in popularity as recording sensors, given that the usage of these devices allows mitigating a number of privacy concerns, a key issue when making use of conventional video cameras. Another concern that is often cited when designing smart home applications is the re…
