Hand gesture recognition systems have yielded many exciting advancements in the last decade and become more popular in HCI (human-computer interaction) with several application areas, which spans from safety and security applications to automotive field. Various deep neural network architectures have already been inspected for hand gesture recognition systems, including multi-layer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN) and a cascade of the last two architectures known as CNN-RNN. However, a major problem still exists, which is most of the existing ML algorithms are designed and developed the building blocks and techniques for real-valued (RV). Researchers applied various RV techniques on the complex-valued (CV) radar images, such as converting a CV optimisation problem into a RV one, by splitting the complex numbers into their real and imaginary parts. However, the major disadvantage of this method is that the resulting algorithm will double the network dimensions. Recent work on RNNs and other fundamental theoretical analysis suggest that CV numbers have a richer representational capacity, but due to the absence of the building blocks required to design such models, the performance of CV networks are marginalised. In this report, we propose a fully CV-CNN, including all building blocks, forward and backward operations, and derivatives all in complex domain. We explore our proposed classification model on two sets of CV hand gesture radar images in comparison with the equivalent RV model. In chapter five, we propose a CV-forward residual network, for the purpose of binary classification of the two sets of CV hand gesture radar datasets and compare its performance with our proposed CV-CNN and a baseline CV-forward CNN.
Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention
KU Leuven(鲁汶大学);Vellore Institute of Technology(韦洛尔理工学院);IMEC(比利时微电子研究中心);Wageningen University and Research(瓦赫宁根大学及研究中心);OnePlanet Research Center(壹行星研究中心)
Hand gesture recognition has long been a hot topic in human computer interaction. Traditional camera-based hand gesture recognition systems cannot work properly under dark circumstances. In this paper, a Doppler Radar based hand gesture recognition system using convolutional neural networks is proposed. A cost-effective Doppler radar sensor with dual receiving channels at 5.8GHz is used to acquire a big database of four standard gestures. The received hand gesture signals are then processed with time-frequency analysis. Convolutional neural networks are used to classify different gestures. Exp…
Ulm University(乌尔姆大学);Otto von Guericke University Magdeburg(马格德堡奥托·冯·格里克大学)
We present a joint camera and radar approach to enable autonomous vehicles to understand and react to human gestures in everyday traffic. Initially, we process the radar data with a PointNet followed by a spatio-temporal multilayer perceptron (stMLP). Independently, the human body pose is extracted from the camera frame and processed with a separate stMLP network. We propose a fusion neural network for both modalities, including an auxiliary loss for each modality. In our experiments with a collected dataset, we show the advantages of gesture recognition with two modalities. Motivated by adver…
Infineon Technologies AG(英飞凌科技公司);Friedrich-Alexander-University Erlangen-Nuremberg(埃尔朗根-纽伦堡弗里德里希-亚历山大大学);Technical University of Munich(慕尼黑工业大学)
Gesture recognition is one of the most intuitive ways of interaction and has gathered particular attention for human computer interaction. Radar sensors possess multiple intrinsic properties, such as their ability to work in low illumination, harsh weather conditions, and being low-cost and compact, making them highly preferable for a gesture recognition solution. However, most literature work focuses on solutions with a limited range that is lower than a meter. We propose a novel architecture for a long-range (1m - 2m) gesture recognition solution that leverages a point cloud-based cross-lear…
Unsupervised Domain Adaptation for RF-based Gesture Recognition
School of Cyber Science and Technology, University of Science and Technology of China(中国科学技术大学网络空间安全学院);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院)
In modern on-driving computing environments, many sensors are used for context-aware applications. This paper utilizes two deep learning models, U-Net and EfficientNet, which consist of a convolutional neural network (CNN), to detect hand gestures and remove noise in the Range Doppler Map image that was measured through a millimeter-wave (mmWave) radar. To improve the performance of classification, accurate pre-processing algorithms are essential. Therefore, a novel pre-processing approach to denoise images before entering the first deep learning model stage increases the accuracy of classific…
Aalto University(阿尔托大学);University of Bergen(卑尔根大学);Aalborg University(奥尔堡大学)
We present Tesla-Rapture, a gesture recognition interface for point clouds generated by mmWave Radars. State of the art gesture recognition models are either too resource consuming or not sufficiently accurate for integration into real-life scenarios using wearable or constrained equipment such as IoT devices (e.g. Raspberry PI), XR hardware (e.g. HoloLens), or smart-phones. To tackle this issue, we developed Tesla, a Message Passing Neural Network (MPNN) graph convolution approach for mmWave radar point clouds. The model outperforms the state of the art on two datasets in terms of accuracy wh…
mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMU
School of Software Engineering, Xi’an Jiaotong University(软件工程学院,西安交通大学);School of Cyber Science and Engineering, Xi’an Jiaotong University(网络科学与工程学院,西安交通大学);School of Computer Science and Technology, Xi’an Jiaotong University(计算机科学与技术学院,西安交通大学);College of Computer Science and Technology, Zhejiang University(计算机科学与技术学院,浙江大学)
Physics-Aware Complex-Valued State Space Model with Scattering-Prior Feature Modulation for PolSAR Image Classification
The College of Information Science and Technology, Beijing University of Chemical Technology(北京化工大学信息科学与技术学院);The School of Information Science and Engineering, Yunnan University(云南大学信息科学与工程学院);The Department of Signal Theory and Communications, Polytechnic University of Catalonia(加泰罗尼亚理工大学信号理论与通信系);The School of Information and Cyberspace Security, Ningxia University(宁夏大学信息与网络空间安全学院)
Live Demonstration: Neuromorphic Radar for Gesture Recognition
Department of Electronic Systems Engineering, Indian Institute of Science, Bangalore, India 560012(电子系统工程系,印度科学院,班加罗尔,印度560012);Department of Electrical Engineering, Indian Institute of Science, Bangalore, India 560012(电气工程系,印度科学院,班加罗尔,印度560012)
Graz University of Technology(格拉茨技术大学);Infineon Technologies Austria AG(英飞凌科技奥地利公司)
Autonomous driving highly depends on capable sensors to perceive the environment and to deliver reliable information to the vehicles' control systems. To increase its robustness, a diversified set of sensors is used, including radar sensors. Radar is a vital contribution of sensory information, providing high resolution range as well as velocity measurements. The increased use of radar sensors in road traffic introduces new challenges. As the so far unregulated frequency band becomes increasingly crowded, radar sensors suffer from mutual interference between multiple radar sensors. This interf…
Graz University of Technology(格拉茨理工大学);Infineon Technologies Austria AG(英飞凌科技奥地利股份公司)
Radar sensors are crucial for environment perception of driver assistance systems as well as autonomous vehicles. With a rising number of radar sensors and the so far unregulated automotive radar frequency band, mutual interference is inevitable and must be dealt with. Algorithms and models operating on radar data are required to run the early processing steps on specialized radar sensor hardware. This specialized hardware typically has strict resource-constraints, i.e. a low memory capacity and low computational power. Convolutional Neural Network (CNN)-based approaches for denoising and inte…
Convolutional neural networks (CNNs) have shown good performance in polarimetric synthetic aperture radar (PolSAR) image classification due to the automation of feature engineering. Excellent hand-crafted architectures of CNNs incorporated the wisdom of human experts, which is an important reason for CNN's success. However, the design of the architectures is a difficult problem, which needs a lot of professional knowledge as well as computational resources. Moreover, the architecture designed by hand might be suboptimal, because it is only one of thousands of unobserved but objective existed p…
Given their substantial success in addressing a wide range of computer vision challenges, Convolutional Neural Networks (CNNs) are increasingly being used in smart home applications, with many of these applications relying on the automatic recognition of human activities. In this context, low-power radar devices have recently gained in popularity as recording sensors, given that the usage of these devices allows mitigating a number of privacy concerns, a key issue when making use of conventional video cameras. Another concern that is often cited when designing smart home applications is the re…
SARLO-80: Worldwide Slant SAR Language Optic Dataset 80cm
DEMR-ONERA – The French Aerospace Lab, Université Paris-Saclay(法国航空航天实验室DEMR-ONERA,巴黎-萨克雷大学);DTIS-ONERA – The French Aerospace Lab, Université Paris-Saclay(法国航空航天实验室DTIS-ONERA,巴黎-萨克雷大学);Hugging Face
CAST: Channel-Aware Spatial Transfer Learning with Pseudo-Image Radar for Sign Language Recognition
Department of Computer Science and Engineering, Khulna University of Engineering & Technology(电子与技术大学计算机科学与工程系);Department of Electrical and Computer Engineering, University of Waterloo(滑铁库大学电子与计算机工程系);Department of Electrical and Computer Engineering, Texas A&M University(德克萨斯A&M大学电子与计算机工程系);College of Applied Computer Science, King Saud University(沙特王后大学应用计算机科学学院);Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence(马尔代夫人工智能大学机器学习系)
Polarimetric synthetic aperture radar (PolSAR) images are widely used in disaster detection and military reconnaissance and so on. However, their interpretation faces some challenges, e.g., deficiency of labeled data, inadequate utilization of data information and so on. In this paper, a complex-valued generative adversarial network (GAN) is proposed for the first time to address these issues. The complex number form of model complies with the physical mechanism of PolSAR data and in favor of utilizing and retaining amplitude and phase information of PolSAR data. GAN architecture and semi-supe…
Convolutional neural networks (CNNs) have been widely used to improve the accuracy of polarimetric synthetic aperture radar (PolSAR) image classification. However, in most studies, the difference between PolSAR images and optical images is rarely considered. Most of the existing CNNs are not tailored for the task of PolSAR image classification, in which complex-valued PolSAR data have been simply equated to real-valued data to fit the optical image processing architectures and avoid complex-valued operations. This is one of the reasons CNNs unable to perform their full capabilities in PolSAR c…
In this paper, we investigated the semantic segmentation of Polarimetric Synthetic Aperture Radar (PolSAR) using Complex-Valued Neural Network (CVNN). Although the coherency matrix is more widely used as the input of CVNN, the Pauli vector has recently been shown to be a valid alternative. We exhaustively compare both methods for six model architectures, three complex-valued, and their respective real-equivalent models. We are comparing, therefore, not only the input representation impact but also the complex- against the real-valued models. We then argue that the dataset splitting produces a …
Deep neural networks have become the main work horse for many tasks involving learning from data in a variety of applications in Science and Engineering. Traditionally, the input to these networks lie in a vector space and the operations employed within the network are well defined on vector-spaces. In the recent past, due to technological advances in sensing, it has become possible to acquire manifold-valued data sets either directly or indirectly. Examples include but are not limited to data from omnidirectional cameras on automobiles, drones etc., synthetic aperture radar imaging, diffusion…
Although complex-valued (CV) neural networks have shown better classification results compared to their real-valued (RV) counterparts for polarimetric synthetic aperture radar (PolSAR) classification, the extension of pixel-level RV networks to the complex domain has not yet thoroughly examined. This paper presents a novel complex-valued deep fully convolutional neural network (CV-FCN) designed for PolSAR image classification. Specifically, CV-FCN uses PolSAR CV data that includes the phase information and utilizes the deep FCN architecture that performs pixel-level labeling. It integrates the…
A Complex-valued SAR Foundation Model Based on Physically Inspired Representation Learning
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息研究院);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院);University of Chinese Academy of Sciences(中国科学院大学)
Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition
School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);Shenzhen Research Institute of Northwestern Polytechnical University(西北工业大学深圳研究院);School of Artificial Intelligence, Chongqing University of Posts and Telecommunications(重庆邮电大学人工智能学院);Aerospace Information Technology University(航天信息大学);Suzhou Aerospace Information Research Institute(苏州航天信息研究所);National Key Laboratory of Microwave Imaging(微波成像国家重点实验室);Aerospace Information Research Institute, CAS(中国科学院航天信息研究所);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子、电气与通信工程学院)