In industrial countries, adults spend a considerable amount of time sedentary each day at work, driving and during activities of daily living. Characterizing the seated upper body human poses using mmWave radars is an important, yet under-studied topic with many applications in human-machine interaction, transportation and road safety. In this work, we devise SUPER, a framework for seated upper body human pose estimation that utilizes dual-mmWave radars in close proximity. A novel masking algorithm is proposed to coherently fuse data from the radars to generate intensity and Doppler point clouds with complementary information for high-motion but small radar cross section areas (e.g., upper extremities) and low-motion but large RCS areas (e.g. torso). A lightweight neural network extracts both global and local features of upper body and output pose parameters for the Skinned Multi-Person Linear (SMPL) model. Extensive leave-one-subject-out experiments on various motion sequences from multiple subjects show that SUPER outperforms a state-of-the-art baseline method by 30 -- 184%. We also demonstrate its utility in a simple downstream task for hand-object interaction.
Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation
School of Computer Science, Peking University（北京大学计算机科学学院）；University of Pittsburgh（匹兹堡大学）；Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)（教育部高可信软件技术重点实验室（北京大学））
RadarNeXt: Real-Time and Reliable 3D Object Detector Based On 4D mmWave Imaging Radar
Institute of Deep Perception Technology, JITRI（捷通华声深度感知技术研究所）；SAT, Xi’an Jiaotong-Liverpool University（西交利物浦大学SAT中心）；University of Liverpool（利物浦大学）；XJTLU-JITRI Academy of Industrial Technology, Xi’an Jiaotong-Liverpool University（西交利物浦大学-捷通华声工业技术研究院）；School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）；Thrust of Artificial Intelligence, HKUST (GZ)（香港科技大学（广州）人工智能学域）；Thrust of Intelligent Transportation, HKUST (GZ)（香港科技大学（广州）智能交通学域）
Chalmers University of Technology（查尔姆斯理工大学）；James Cook University（詹姆斯库克大学）；La Trobe University（乐卓博大学）；Beihang University（北京航空航天大学）；Vitalent Consulting（Vitalent咨询公司）；Silo AI
The automotive mmWave radar plays a key role in advanced driver assistance systems (ADAS) and autonomous driving. Deep learning-based instance segmentation enables real-time object identification from the radar detection points. In the conventional training process, accurate annotation is the key. However, high-quality annotations of radar detection points are challenging to achieve due to their ambiguity and sparsity. To address this issue, we propose a contrastive learning approach for implementing radar detection points-based instance segmentation. We define the positive and negative sample…
Carnegie Mellon University（卡内基梅隆大学）；Bosch Research and Technology Center（博世研究与技术中心）
mmWave radars offer excellent depth resolution even at very long ranges owing to their high bandwidth. But their angular resolution is at least an order-of-magnitude worse than camera and lidar systems. Hence, mmWave radar is not a capable 3-D imaging solution in isolation. We propose Metamoran, a system that combines the complimentary strengths of radar and camera to obtain accurate, high resolution depth images over long ranges even in high clutter environments, all from a single fixed vantage point. Metamoran enables rich long-range depth imaging with applications in security and surveillan…
In this paper we presented mmPose-NLP, a novel Natural Language Processing (NLP) inspired Sequence-to-Sequence (Seq2Seq) skeletal key-point estimator using millimeter-wave (mmWave) radar data. To the best of the author's knowledge, this is the first method to precisely estimate upto 25 skeletal key-points using mmWave radar data alone. Skeletal pose estimation is critical in several applications ranging from autonomous vehicles, traffic monitoring, patient monitoring, gait analysis, to defense security forensics, and aid both preventative and actionable decision making. The use of mmWave radar…
In modern on-driving computing environments, many sensors are used for context-aware applications. This paper utilizes two deep learning models, U-Net and EfficientNet, which consist of a convolutional neural network (CNN), to detect hand gestures and remove noise in the Range Doppler Map image that was measured through a millimeter-wave (mmWave) radar. To improve the performance of classification, accurate pre-processing algorithms are essential. Therefore, a novel pre-processing approach to denoise images before entering the first deep learning model stage increases the accuracy of classific…
Since electromagnetic signals are omnipresent, Radio Frequency (RF)-sensing has the potential to become a universal sensing mechanism with applications in localization, smart-home, retail, gesture recognition, intrusion detection, etc. Two emerging technologies in RF-sensing, namely sensing through Large Intelligent Surfaces (LISs) and mmWave Frequency-Modulated Continuous-Wave (FMCW) radars, have been successfully applied to a wide range of applications. In this work, we compare LIS and mmWave radars for localization in real-world and simulated environments. In our experiments, the mmWave rad…
In this paper, we explore the feasibility of utilizing a mmWave radar sensor installed on a UAV to reconstruct the 3D shapes of multiple objects in a space. The UAV hovers at various locations in the space, and its onboard radar senor collects raw radar data via scanning the space with Synthetic Aperture Radar (SAR) operation. The radar data is sent to a deep neural network model, which outputs the point cloud reconstruction of the multiple objects in the space. We evaluate two different models. Model 1 is our recently proposed 3DRIMR/R2P model, and Model 2 is formed by adding a segmentation s…
Beijing University of Posts and Telecommunications（北京邮电大学）
With autonomous driving developing in a booming stage, accurate object detection in complex scenarios attract wide attention to ensure the safety of autonomous driving. Millimeter wave (mmWave) radar and vision fusion is a mainstream solution for accurate obstacle detection. This article presents a detailed survey on mmWave radar and vision fusion based obstacle detection methods. First, we introduce the tasks, evaluation criteria, and datasets of object detection for autonomous driving. The process of mmWave radar and vision fusion is then divided into three parts: sensor deployment, sensor c…
Recent research has shown the effectiveness of mmWave radar sensing for object detection in low visibility environments, which makes it an ideal technique in autonomous navigation systems. In this paper, we introduce Radar to Point Cloud (R2P), a deep learning model that generates smooth, dense, and highly accurate point cloud representation of a 3D object with fine geometry details, based on rough and sparse point clouds with incorrect points obtained from mmWave radar. These input point clouds are converted from the 2D depth images that are generated from raw mmWave radar sensor data, charac…
mmWave radar has been shown as an effective sensing technique in low visibility, smoke, dusty, and dense fog environment. However tapping the potential of radar sensing to reconstruct 3D object shapes remains a great challenge, due to the characteristics of radar data such as sparsity, low resolution, specularity, high noise, and multi-path induced shadow reflections and artifacts. In this paper we propose 3D Reconstruction and Imaging via mmWave Radar (3DRIMR), a deep learning based architecture that reconstructs 3D shape of an object in dense detailed point cloud format, based on sparse raw …
ProbRadarM3F: mmWave Radar based Human Skeletal Pose Estimation with Probability Map Guided Multi-Format Feature Fusion
School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, P.R.~China（北京航空航天大学自动化科学与电气工程学院）；School of Electronics and Information Engineering, Beihang University, Beijing 100191, P.R.~China（北京航空航天大学电子信息工程学院）；College of Science and Engineering, James Cook University, Cairns, Australia（詹姆斯库克大学科学与工程学院）；School of Computing, Engineering and Mathematical Sciences, La Trobe University, Melbourne, Australia（拉特罗布大学计算、工程与数学科学学院）
VISC: mmWave Radar Scene Flow Estimation using Pervasive Visual-Inertial Supervision
State Key Laboratory of Maritime Technology and Safety（maritime技术与安全国家重点实验室）；Wuhan University of Technology（武汉理工大学）；University of Essex（埃塞克斯大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）；University College London（伦敦大学学院）
Aalto University（阿尔托大学）；University of Bergen（卑尔根大学）；Aalborg University（奥尔堡大学）
We present Tesla-Rapture, a gesture recognition interface for point clouds generated by mmWave Radars. State of the art gesture recognition models are either too resource consuming or not sufficiently accurate for integration into real-life scenarios using wearable or constrained equipment such as IoT devices (e.g. Raspberry PI), XR hardware (e.g. HoloLens), or smart-phones. To tackle this issue, we developed Tesla, a Message Passing Neural Network (MPNN) graph convolution approach for mmWave radar point clouds. The model outperforms the state of the art on two datasets in terms of accuracy wh…
Ultra-High-Frequency Harmony: mmWave Radar and Event Camera Orchestrate Accurate Drone Landing
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Carnegie Mellon University（卡内基梅隆大学）；Meituan Academy of Robotics Shenzhen（美团深圳机器人研究院）；School of Software, Tsinghua University（清华大学软件学院）；Pengcheng Laboratory（鹏城实验室）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）
Equi-RO: A 4D mmWave Radar Odometry via Equivariant Networks
School of Vehicle and Mobility, Tsinghua University（车辆与移动系统学院，清华大学）；Computational Autonomy and Robotic Laboratory, University of Michigan（计算自主与机器人实验室，密歇根大学）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（智能绿色车辆与移动系统国家重点实验室，清华大学）
Southeast University（东南大学）；University of Minnesota（明尼苏达大学）；Singapore University of Technology and Design（新加坡科技设计大学）；The University of Edinburgh（爱丁堡大学）
Human identification is a key requirement for many applications in everyday life, such as personalized services, automatic surveillance, continuous authentication, and contact tracing during pandemics, etc. This work studies the problem of cross-modal human re-identification (ReID), in response to the regular human movements across camera-allowed regions (e.g., streets) and camera-restricted regions (e.g., offices) deployed with heterogeneous sensors. By leveraging the emerging low-cost RGB-D cameras and mmWave radars, we propose the first-of-its-kind vision-RF system for cross-modal multi-per…
DAP: Doppler-aware Point Network for Heterogeneous mmWave Action Recognition
Peking University（北京大学）；Huazhong University of Science and Technology（华中科技大学）；DJI Technology Company Ltd.（大疆技术创新有限公司）；Christian-Albrechts-Universität zu Kiel（基尔大学）
Exploring Spatial-Temporal Representation via Star Graph for mmWave Radar-based Human Activity Recognition
School of Computer Science and Informatics, University of Liverpool（计算机科学与信息学系，利物浦大学）；School of Computer Science and Engineering, Southeast University（计算机科学与工程系，东南大学）；City University of Hong Kong（香港城市大学）；Knight Foundation School of Computing and Information Sciences, Florida International University（Knight基金会计算与信息科学系，佛罗里达国际大学）
UMass Boston（马萨诸塞大学波士顿分校）；UMass Lowell（马萨诸塞大学洛厄尔分校）；Department of Computer Science, UMass Boston（马萨诸塞大学波士顿分校计算机科学系）；Department of Engineering, UMass Boston（马萨诸塞大学波士顿分校工程系）；Department of Computer Science, UMass Lowell（马萨诸塞大学洛厄尔分校计算机科学系）；UMass Boston, USA（美国马萨诸塞大学波士顿分校）；UMass Lowell, USA（美国马萨诸塞大学洛厄尔分校）；Department of Computer Science（计算机科学系）；Department of Engineering（工程系）
Recent research has shown that mmWave radar sensing is effective for object detection in low visibility environments, which makes it an ideal technique in autonomous navigation systems such as autonomous vehicles. However, due to the characteristics of radar signals such as sparsity, low resolution, specularity, and high noise, it is still quite challenging to reconstruct 3D object shapes via mmWave radar sensing. Built on our recent proposed 3DRIMR (3D Reconstruction and Imaging via mmWave Radar), we introduce in this paper DeepPoint, a deep learning model that generates 3D objects in point c…
mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual Reality
Southeast University（东南大学）；City University of Hong Kong（香港城市大学）；Michigan State University（密歇根州立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Minnesota（明尼苏达大学）
HybridSim: A Physics-Learning Hybrid Digital Twin for mmWave Human Sensing
Xiamen University Malaysia（厦门大学马来西亚分校）；The Hong Kong University of Science and Technology（香港科技大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Zhejiang University（浙江大学）；Key Laboratory of Collaborative Sensing and Autonomous Unmanned Systems of Zhejiang Province（浙江省协同感知与自主无人系统重点实验室）
Gait recognition is widely used in diversified practical applications. Currently, the most prevalent approach is to recognize human gait from RGB images, owing to the progress of computer vision technologies. Nevertheless, the perception capability of RGB cameras deteriorates in rough circumstances, and visual surveillance may cause privacy invasion. Due to the robustness and non-invasive feature of millimeter wave (mmWave) radar, radar-based gait recognition has attracted increasing attention in recent years. In this research, we propose a Hierarchical Dynamic Network (HDNet) for gait recogni…
Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension
Institute of Deep Perception Technology, JITRI（JITRI深度感知技术研究所）；University of Liverpool（利物浦大学）；University of Southampton（南安普顿大学）；Momoni AI（Momoni人工智能公司）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Thrust of Artificial Intelligence, HKUST (GZ)（香港科技大学（广州）人工智能学域）；Thrust of Intelligent Transportation, HKUST (GZ)（香港科技大学（广州）智能交通学域）
mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMU
School of Software Engineering, Xi’an Jiaotong University（软件工程学院，西安交通大学）；School of Cyber Science and Engineering, Xi’an Jiaotong University（网络科学与工程学院，西安交通大学）；School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学）；College of Computer Science and Technology, Zhejiang University（计算机科学与技术学院，浙江大学）
