In modern on-driving computing environments, many sensors are used for context-aware applications. This paper utilizes two deep learning models, U-Net and EfficientNet, which consist of a convolutional neural network (CNN), to detect hand gestures and remove noise in the Range Doppler Map image that was measured through a millimeter-wave (mmWave) radar. To improve the performance of classification, accurate pre-processing algorithms are essential. Therefore, a novel pre-processing approach to denoise images before entering the first deep learning model stage increases the accuracy of classification. Thus, this paper proposes a deep neural network based high-performance nonlinear pre-processing method.
In this paper we presented mmPose-NLP, a novel Natural Language Processing (NLP) inspired Sequence-to-Sequence (Seq2Seq) skeletal key-point estimator using millimeter-wave (mmWave) radar data. To the best of the author's knowledge, this is the first method to precisely estimate upto 25 skeletal key-points using mmWave radar data alone. Skeletal pose estimation is critical in several applications ranging from autonomous vehicles, traffic monitoring, patient monitoring, gait analysis, to defense security forensics, and aid both preventative and actionable decision making. The use of mmWave radar…
Since electromagnetic signals are omnipresent, Radio Frequency (RF)-sensing has the potential to become a universal sensing mechanism with applications in localization, smart-home, retail, gesture recognition, intrusion detection, etc. Two emerging technologies in RF-sensing, namely sensing through Large Intelligent Surfaces (LISs) and mmWave Frequency-Modulated Continuous-Wave (FMCW) radars, have been successfully applied to a wide range of applications. In this work, we compare LIS and mmWave radars for localization in real-world and simulated environments. In our experiments, the mmWave rad…
In this paper, we present a 2D target localization method using two low cost and compact Mellimeter Wave Frequency Modulated Continuous Wave (MMW-FMCW) radars. To create a 2D map we exploit the bilateration method followed by a multi-target tracking block to remove the ghost targets. Finally, we present experimental results based on the data gathered from two FMCW radars operating at $\rm 79 \;GHz$.
In this paper, we explore the feasibility of utilizing a mmWave radar sensor installed on a UAV to reconstruct the 3D shapes of multiple objects in a space. The UAV hovers at various locations in the space, and its onboard radar senor collects raw radar data via scanning the space with Synthetic Aperture Radar (SAR) operation. The radar data is sent to a deep neural network model, which outputs the point cloud reconstruction of the multiple objects in the space. We evaluate two different models. Model 1 is our recently proposed 3DRIMR/R2P model, and Model 2 is formed by adding a segmentation s…
The wide bandwidths available at millimeter-wave (mmWave) frequencies have offered exciting potential to wireless communication systems and radar alike. Communication systems can offer higher rates and support more users with mmWave bands while radar systems can benefit from higher resolution captures. This leads to the possibility that portions of mmWave spectrum will be occupied by both communication and radar (e.g., 60 GHz industrial, scientific, and medical (ISM) band). This potential coexistence motivates the work of this paper, in which we present a design that can enable simultaneous, i…
Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention
KU Leuven（鲁汶大学）；Vellore Institute of Technology（韦洛尔理工学院）；IMEC（比利时微电子研究中心）；Wageningen University and Research（瓦赫宁根大学及研究中心）；OnePlanet Research Center（壹行星研究中心）
The paper proposes a novel feature extraction approach for FMCW radar systems in the field of short-range gesture sensing. A light-weight processing is proposed which reduces a series of 3D radar data cubes to four 1D time signals containing information about range, azimuth angle, elevation angle and magnitude. The processing is entirely performed in the time domain without using any Fourier transformation and enables the training of a deep neural network directly on the raw time domain data. It is shown experimentally on real world data, that the proposed processing retains the same expressiv…
As a cost-effective alternative, hybrid analog and digital beamforming architecture is a promising scheme for millimeter wave (mmWave) system. This paper considers two hybrid beamforming architectures, i.e. the partially-connected and fully-connected structures, for mmWave dual-function radar communication (DFRC) system, where the transmitter communicates with the downlink users and detects radar targets simultaneously. The optimization problems are formulated by minimizing a weighted summation of radar and communication performance, subject to constant modulus and power constraints. To tackle…
We introduce a simple but effective technique in automatic hand gesture recognition using radar. The proposed technique classifies hand gestures based on the envelopes of their micro-Doppler signatures. These envelopes capture the distinctions among different hand movements and their corresponding positive and negative Doppler frequencies which are generated during each gesture act. We detect the positive and negative envelopes separately, and form a feature vector of their augmentation. We use the $k$-nearest neighbor ($k$NN) classifier and Manhattan distance (L1) measure, in lieu of Euclidea…
Millimeter wave sensing has recently attracted a lot of attention given its environmental robust nature. In situations where visual sensors like cameras fail to perform, mmwave radars can be used to achieve reliable performance. However, because of the poor scattering performance and lack of texture in millimeter waves, radars can not be used in several situations that require precise identification of objects. In this paper, we take insight from camera fiducials which are very easily identifiable by the camera, and present R-fiducial tags, which smartly augment the current infrastructure to e…
Radar processing via spiking neural networks (SNNs) has recently emerged as a solution in the field of ultra-low-power wireless human-computer interaction. Compared to traditional energy- and area-hungry deep learning methods, SNNs are significantly more energy efficient and can be deployed in the growing number of compact SNN accelerator chips, making them a better solution for ubiquitous IoT applications. We propose a novel SNN strategy for radar gesture recognition, achieving more than 91% of accuracy on two different radar datasets. Our work significantly differs from previous approaches a…
This paper presents the design, implementation and evaluation of milliMap, a single-chip millimetre wave (mmWave) radar based indoor mapping system targetted towards low-visibility environments to assist in emergency response. A unique feature of milliMap is that it only leverages a low-cost, off-the-shelf mmWave radar, but can reconstruct a dense grid map with accuracy comparable to lidar, as well as providing semantic annotations of objects on the map. milliMap makes two key technical contributions. First, it autonomously overcomes the sparsity and multi-path noise of mmWave signals by combi…
Efficient Dual-Blind Deconvolution for Joint Radar-Communication Systems Using ADMM: Enhancing Channel Estimation and Signal Recovery in 5G mmWave Networks
Millimeter wave (mmWave) based gesture recognition technology provides a good human computer interaction (HCI) experience. Prior works focus on the close-range gesture recognition, but fall short in range extension, i.e., they are unable to recognize gestures more than one meter away from considerable noise motions. In this paper, we design a long-range gesture recognition model which utilizes a novel data processing method and a customized artificial Convolutional Neural Network (CNN). Firstly, we break down gestures into multiple reflection points and extract their spatial-temporal features …
Millimeter-wave radars are being increasingly integrated into commercial vehicles to support advanced driver-assistance system features. A key shortcoming for present-day vehicular radar imaging is poor azimuth resolution (for side-looking operation) due to the form factor limits on antenna size and placement. In this paper, we propose a solution via a new multiple-input and multiple-output synthetic aperture radar (MIMO-SAR) imaging technique, that applies coherent SAR principles to vehicular MIMO radar to improve the side-view (angular) resolution. The proposed 2-stage hierarchical MIMO-SAR …
In this paper, we propose a beamforming design for dual-functional radar-communication (DFRC) systems at the millimeter wave (mmWave) band, where hybrid beamforming and sub-arrayed MIMO radar techniques are jointly exploited. We assume that a base station (BS) is serving a user equipment (UE) located in a Non-Line-of-Sight (NLoS) channel, which in the meantime actively detects multiple targets located in a Line-of-Sight (LoS) channel. Given the optimal communication beamformer and the desired radar beampattern, we propose to design the analog and digital beamformers under non-convex constant-m…
In this paper, a real-time signal processing frame-work based on a 60 GHz frequency-modulated continuous wave (FMCW) radar system to recognize gestures is proposed. In order to improve the robustness of the radar-based gesture recognition system, the proposed framework extracts a comprehensive hand profile, including range, Doppler, azimuth and elevation, over multiple measurement-cycles and encodes them into a feature cube. Rather than feeding the range-Doppler spectrum sequence into a deep convolutional neural network (CNN) connected with recurrent neural networks, the proposed framework tak…
Millimeter-Wave (mmWave) radar sensors are gaining popularity for their robust sensing and increasing imaging capabilities. However, current radar signal processing is hardware specific, which makes it impossible to build sensor agnostic solutions. OpenRadar serves as an interface to prototype, research, and benchmark solutions in a modular manner. This enables creating software processing stacks in a way that has not yet been extensively explored. In the wake of increased AI adoption, OpenRadar can accelerate the growth of the combined fields of radar and AI. The OpenRadar API was released on…
FMCW radar could detect object's range, speed and Angleof-Arrival, advantages are robust to bad weather, good range resolution, and good speed resolution. In this paper, we consider the FMCW radar as a novel interacting interface on laptop. We merge sequences of object's range, speed, azimuth information into single input, then feed to a convolution neural network to learn spatial and temporal patterns. Our model achieved 96% accuracy on test set and real-time test.
Infineon Technologies AG（英飞凌科技公司）；Friedrich-Alexander-University Erlangen-Nuremberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Technical University of Munich（慕尼黑工业大学）
Gesture recognition is one of the most intuitive ways of interaction and has gathered particular attention for human computer interaction. Radar sensors possess multiple intrinsic properties, such as their ability to work in low illumination, harsh weather conditions, and being low-cost and compact, making them highly preferable for a gesture recognition solution. However, most literature work focuses on solutions with a limited range that is lower than a meter. We propose a novel architecture for a long-range (1m - 2m) gesture recognition solution that leverages a point cloud-based cross-lear…
Large-scale deployment of connected vehicles with cooperative sensing and maneuvering technologies increases the demand for vehicle-to-everything communication (V2X) band in 5.9 GHz. Besides the V2X spectrum, the under-utilized millimeter-wave (mmWave) bands at 24 and 77 GHz can be leveraged to supplement V2X communication and support high data rates for emerging broadband applications. For this purpose, joint radar-communication (JRC) systems have been proposed in the literature to perform both functions using the same waveform and hardware. In this work, we present a software-defined multipl…
mmWave radars have recently gathered significant attention as a means to track human movement within indoor environments. Widely adopted Kalman filter tracking methods experience performance degradation when the underlying movement is highly non-linear or presents long-term temporal dependencies. As a solution, in this article we design a convolutional-recurrent Neural Network (NN) that learns to accurately estimate the position and the velocity of the monitored subjects from high dimensional radar data. The NN is trained as a probabilistic model, utilizing a Gaussian negative log-likelihood l…
The high overhead of the beam training process is the main challenge when establishing mmWave communication links, especially for vehicle-to-everything (V2X) scenarios where the channels are highly dynamic. In this paper, we obtain prior information to speed up the beam training process by implementing two deep neural networks (DNNs) that realize radar-to-communication (R2C) channel information translation in a vehicle-to-infrastructure (V2I) system. Specifically, the first DNN is built to extract the information from the radar azimuth power spectrum (APS) to reconstruct the communication APS,…
This paper explores an integrated approach for improved sensing and positioning with applications in air traffic management (ATM) and in the Advanced Surface Movement Guidance and Control System (A-SMGCS). The integrated approach includes the synergy of 3D Vector Antenna with the novel time-of-arrival and angle-of-arrival estimate methods for accurate positioning, combining the sensing on the sub-6GHz and mmWave spectrum for the enhanced non-cooperative surveillance. For the positioning scope, both uplink and downlink 5G reference signals are investigated and their performance is evaluated. Fo…
University of Washington（华盛顿大学）；Silkwave Holdings（矽控控股）
Millimeter-wave (mmW) radars are being increasingly integrated in commercial vehicles to support new Adaptive Driver Assisted Systems (ADAS) for its ability to provide high accuracy location, velocity, and angle estimates of objects, largely independent of environmental conditions. Such radar sensors not only perform basic functions such as detection and ranging/angular localization, but also provide critical inputs for environmental perception via object recognition and classification. To explore radar-based ADAS applications, we have assembled a lab-scale frequency modulated continuous wave …
Continuous monitoring of human vital signs using non-contact mmWave radars is attractive due to their ability to penetrate garments and operate under different lighting conditions. Unfortunately, most prior research requires subjects to stay at a fixed distance from radar sensors and to remain still during monitoring. These restrictions limit the applications of radar vital sign monitoring in real life scenarios. In this paper, we address these limitations and present "Pi-ViMo", a non-contact Physiology-inspired Robust Vital Sign Monitoring system, using mmWave radars. We first derive a multi-…
Robust Multimodal Learning Framework For Intake Gesture Detection Using Contactless Radar and Wearable IMU Sensors
e-Media Research Lab（e-Media研究实验室）；ESAT-STADIUS Division, KU Leuven（ESAT-STADIUS部门，KU莱顿大学）；M-Group, DistriNet, Department of Computer Science, KU Leuven（M组、DistriNet、计算机科学系，KU莱顿大学）
University of Oxford（牛津大学）；University of Edinburgh（爱丁堡大学）
mmWave FMCW radar has attracted huge amount of research interest for human-centered applications in recent years, such as human gesture/activity recognition. Most existing pipelines are built upon conventional Discrete Fourier Transform (DFT) pre-processing and deep neural network classifier hybrid methods, with a majority of previous works focusing on designing the downstream classifier to improve overall accuracy. In this work, we take a step back and look at the pre-processing module. To avoid the drawbacks of conventional DFT pre-processing, we propose a learnable pre-processing module, na…
4D Millimeter-Wave Radar in Autonomous Driving: A Survey
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）；Xingjian College, Tsinghua University（清华大学行健书院）；Changan Automobile Co., Ltd.（长安汽车股份有限公司）
Configuring millimeter wave links following a conventional beam training protocol, as the one proposed in the current cellular standard, introduces a large communication overhead, specially relevant in vehicular systems, where the channels are highly dynamic. In this paper, we propose the use of a passive radar array to sense automotive radar transmissions coming from multiple vehicles on the road, and a radar processing chain that provides information about a reduced set of candidate beams for the links between the road-infrastructure and each one of the vehicles. This prior information can b…
University of Moratuwa（莫拉图瓦大学）；Northeastern University（东北大学）；Florida International University（佛罗里达国际大学）
Transformers combined with convolutional encoders have been recently used for hand gesture recognition (HGR) using micro-Doppler signatures. We propose a vision-transformer-based architecture for HGR with multi-antenna continuous-wave Doppler radar receivers. The proposed architecture consists of three modules: a convolutional encoderdecoder, an attention module with three transformer layers, and a multi-layer perceptron. The novel convolutional decoder helps to feed patches with larger sizes to the attention module for improved feature extraction. Experimental results obtained with a dataset …
