自动驾驶借助人工智能与传感器技术，在解决交通安全问题方面极具前景。多目标跟踪（Multi-Object Tracking）是保障车辆在复杂交通场景下安全高效导航的关键环节。本文提出一种新型深度学习方法，融合雷达与相机数据，提升自动驾驶系统中多目标跟踪的准确性与鲁棒性。该方法采用双向长短期记忆网络（Bi-directional Long Short-Term Memory network）引入长时时序信息，优化运动预测；使用受FaceNet启发的外观特征模型建立不同帧间的目标关联，确保跟踪一致性；采用三输出机制，包含雷达、相机各自的独立输出及融合输出，可应对传感器故障并输出准确跟踪结果。通过在真实数据集上开展大量评估，该方法在跟踪精度上实现显著提升，即便在低能见度场景下也能保持可靠性能。
英文摘要：
Autonomous driving holds great promise in addressing traffic safety concerns by leveraging artificial intelligence and sensor technology. Multi-Object Tracking plays a critical role in ensuring safer and more efficient navigation through complex traffic scenarios. This paper presents a novel deep learning-based method that integrates radar and camera data to enhance the accuracy and robustness of Multi-Object Tracking in autonomous driving systems. The proposed method leverages a Bi-directional Long Short-Term Memory network to incorporate long-term temporal information and improve motion prediction. An appearance feature model inspired by FaceNet is used to establish associations between objects across different frames, ensuring consistent tracking. A tri-output mechanism is employed, consisting of individual outputs for radar and camera sensors and a fusion output, to provide robustness against sensor failures and produce accurate tracking results. Through extensive evaluations of real-world datasets, our approach demonstrates remarkable improvements in tracking accuracy, ensuring reliable performance even in low-visibility scenarios.
DAP: Doppler-aware Point Network for Heterogeneous mmWave Action Recognition
Peking University（北京大学）；Huazhong University of Science and Technology（华中科技大学）；DJI Technology Company Ltd.（大疆技术创新有限公司）；Christian-Albrechts-Universität zu Kiel（基尔大学）
mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMU
School of Software Engineering, Xi’an Jiaotong University（软件工程学院，西安交通大学）；School of Cyber Science and Engineering, Xi’an Jiaotong University（网络科学与工程学院，西安交通大学）；School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学）；College of Computer Science and Technology, Zhejiang University（计算机科学与技术学院，浙江大学）
Since electromagnetic signals are omnipresent, Radio Frequency (RF)-sensing has the potential to become a universal sensing mechanism with applications in localization, smart-home, retail, gesture recognition, intrusion detection, etc. Two emerging technologies in RF-sensing, namely sensing through Large Intelligent Surfaces (LISs) and mmWave Frequency-Modulated Continuous-Wave (FMCW) radars, have been successfully applied to a wide range of applications. In this work, we compare LIS and mmWave radars for localization in real-world and simulated environments. In our experiments, the mmWave rad…
In this paper we presented mmPose-NLP, a novel Natural Language Processing (NLP) inspired Sequence-to-Sequence (Seq2Seq) skeletal key-point estimator using millimeter-wave (mmWave) radar data. To the best of the author's knowledge, this is the first method to precisely estimate upto 25 skeletal key-points using mmWave radar data alone. Skeletal pose estimation is critical in several applications ranging from autonomous vehicles, traffic monitoring, patient monitoring, gait analysis, to defense security forensics, and aid both preventative and actionable decision making. The use of mmWave radar…
In modern on-driving computing environments, many sensors are used for context-aware applications. This paper utilizes two deep learning models, U-Net and EfficientNet, which consist of a convolutional neural network (CNN), to detect hand gestures and remove noise in the Range Doppler Map image that was measured through a millimeter-wave (mmWave) radar. To improve the performance of classification, accurate pre-processing algorithms are essential. Therefore, a novel pre-processing approach to denoise images before entering the first deep learning model stage increases the accuracy of classific…
In this paper, we explore the feasibility of utilizing a mmWave radar sensor installed on a UAV to reconstruct the 3D shapes of multiple objects in a space. The UAV hovers at various locations in the space, and its onboard radar senor collects raw radar data via scanning the space with Synthetic Aperture Radar (SAR) operation. The radar data is sent to a deep neural network model, which outputs the point cloud reconstruction of the multiple objects in the space. We evaluate two different models. Model 1 is our recently proposed 3DRIMR/R2P model, and Model 2 is formed by adding a segmentation s…
Beijing University of Posts and Telecommunications（北京邮电大学）
With autonomous driving developing in a booming stage, accurate object detection in complex scenarios attract wide attention to ensure the safety of autonomous driving. Millimeter wave (mmWave) radar and vision fusion is a mainstream solution for accurate obstacle detection. This article presents a detailed survey on mmWave radar and vision fusion based obstacle detection methods. First, we introduce the tasks, evaluation criteria, and datasets of object detection for autonomous driving. The process of mmWave radar and vision fusion is then divided into three parts: sensor deployment, sensor c…
Recent research has shown the effectiveness of mmWave radar sensing for object detection in low visibility environments, which makes it an ideal technique in autonomous navigation systems. In this paper, we introduce Radar to Point Cloud (R2P), a deep learning model that generates smooth, dense, and highly accurate point cloud representation of a 3D object with fine geometry details, based on rough and sparse point clouds with incorrect points obtained from mmWave radar. These input point clouds are converted from the 2D depth images that are generated from raw mmWave radar sensor data, charac…
mmWave radar has been shown as an effective sensing technique in low visibility, smoke, dusty, and dense fog environment. However tapping the potential of radar sensing to reconstruct 3D object shapes remains a great challenge, due to the characteristics of radar data such as sparsity, low resolution, specularity, high noise, and multi-path induced shadow reflections and artifacts. In this paper we propose 3D Reconstruction and Imaging via mmWave Radar (3DRIMR), a deep learning based architecture that reconstructs 3D shape of an object in dense detailed point cloud format, based on sparse raw …
Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation
School of Computer Science, Peking University（北京大学计算机科学学院）；University of Pittsburgh（匹兹堡大学）；Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)（教育部高可信软件技术重点实验室（北京大学））
VISC: mmWave Radar Scene Flow Estimation using Pervasive Visual-Inertial Supervision
State Key Laboratory of Maritime Technology and Safety（maritime技术与安全国家重点实验室）；Wuhan University of Technology（武汉理工大学）；University of Essex（埃塞克斯大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）；University College London（伦敦大学学院）
RadarNeXt: Real-Time and Reliable 3D Object Detector Based On 4D mmWave Imaging Radar
Institute of Deep Perception Technology, JITRI（捷通华声深度感知技术研究所）；SAT, Xi’an Jiaotong-Liverpool University（西交利物浦大学SAT中心）；University of Liverpool（利物浦大学）；XJTLU-JITRI Academy of Industrial Technology, Xi’an Jiaotong-Liverpool University（西交利物浦大学-捷通华声工业技术研究院）；School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）；Thrust of Artificial Intelligence, HKUST (GZ)（香港科技大学（广州）人工智能学域）；Thrust of Intelligent Transportation, HKUST (GZ)（香港科技大学（广州）智能交通学域）
ProbRadarM3F: mmWave Radar based Human Skeletal Pose Estimation with Probability Map Guided Multi-Format Feature Fusion
School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, P.R.~China（北京航空航天大学自动化科学与电气工程学院）；School of Electronics and Information Engineering, Beihang University, Beijing 100191, P.R.~China（北京航空航天大学电子信息工程学院）；College of Science and Engineering, James Cook University, Cairns, Australia（詹姆斯库克大学科学与工程学院）；School of Computing, Engineering and Mathematical Sciences, La Trobe University, Melbourne, Australia（拉特罗布大学计算、工程与数学科学学院）
Aalto University（阿尔托大学）；University of Bergen（卑尔根大学）；Aalborg University（奥尔堡大学）
We present Tesla-Rapture, a gesture recognition interface for point clouds generated by mmWave Radars. State of the art gesture recognition models are either too resource consuming or not sufficiently accurate for integration into real-life scenarios using wearable or constrained equipment such as IoT devices (e.g. Raspberry PI), XR hardware (e.g. HoloLens), or smart-phones. To tackle this issue, we developed Tesla, a Message Passing Neural Network (MPNN) graph convolution approach for mmWave radar point clouds. The model outperforms the state of the art on two datasets in terms of accuracy wh…
Chalmers University of Technology（查尔姆斯理工大学）；James Cook University（詹姆斯库克大学）；La Trobe University（乐卓博大学）；Beihang University（北京航空航天大学）；Vitalent Consulting（Vitalent咨询公司）；Silo AI
The automotive mmWave radar plays a key role in advanced driver assistance systems (ADAS) and autonomous driving. Deep learning-based instance segmentation enables real-time object identification from the radar detection points. In the conventional training process, accurate annotation is the key. However, high-quality annotations of radar detection points are challenging to achieve due to their ambiguity and sparsity. To address this issue, we propose a contrastive learning approach for implementing radar detection points-based instance segmentation. We define the positive and negative sample…
Ultra-High-Frequency Harmony: mmWave Radar and Event Camera Orchestrate Accurate Drone Landing
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Carnegie Mellon University（卡内基梅隆大学）；Meituan Academy of Robotics Shenzhen（美团深圳机器人研究院）；School of Software, Tsinghua University（清华大学软件学院）；Pengcheng Laboratory（鹏城实验室）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）
Equi-RO: A 4D mmWave Radar Odometry via Equivariant Networks
School of Vehicle and Mobility, Tsinghua University（车辆与移动系统学院，清华大学）；Computational Autonomy and Robotic Laboratory, University of Michigan（计算自主与机器人实验室，密歇根大学）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（智能绿色车辆与移动系统国家重点实验室，清华大学）
Exploring Spatial-Temporal Representation via Star Graph for mmWave Radar-based Human Activity Recognition
School of Computer Science and Informatics, University of Liverpool（计算机科学与信息学系，利物浦大学）；School of Computer Science and Engineering, Southeast University（计算机科学与工程系，东南大学）；City University of Hong Kong（香港城市大学）；Knight Foundation School of Computing and Information Sciences, Florida International University（Knight基金会计算与信息科学系，佛罗里达国际大学）
UMass Boston（马萨诸塞大学波士顿分校）；UMass Lowell（马萨诸塞大学洛厄尔分校）；Department of Computer Science, UMass Boston（马萨诸塞大学波士顿分校计算机科学系）；Department of Engineering, UMass Boston（马萨诸塞大学波士顿分校工程系）；Department of Computer Science, UMass Lowell（马萨诸塞大学洛厄尔分校计算机科学系）；UMass Boston, USA（美国马萨诸塞大学波士顿分校）；UMass Lowell, USA（美国马萨诸塞大学洛厄尔分校）；Department of Computer Science（计算机科学系）；Department of Engineering（工程系）
Recent research has shown that mmWave radar sensing is effective for object detection in low visibility environments, which makes it an ideal technique in autonomous navigation systems such as autonomous vehicles. However, due to the characteristics of radar signals such as sparsity, low resolution, specularity, and high noise, it is still quite challenging to reconstruct 3D object shapes via mmWave radar sensing. Built on our recent proposed 3DRIMR (3D Reconstruction and Imaging via mmWave Radar), we introduce in this paper DeepPoint, a deep learning model that generates 3D objects in point c…
Carnegie Mellon University（卡内基梅隆大学）；Bosch Research and Technology Center（博世研究与技术中心）
mmWave radars offer excellent depth resolution even at very long ranges owing to their high bandwidth. But their angular resolution is at least an order-of-magnitude worse than camera and lidar systems. Hence, mmWave radar is not a capable 3-D imaging solution in isolation. We propose Metamoran, a system that combines the complimentary strengths of radar and camera to obtain accurate, high resolution depth images over long ranges even in high clutter environments, all from a single fixed vantage point. Metamoran enables rich long-range depth imaging with applications in security and surveillan…
Southeast University（东南大学）；University of Minnesota（明尼苏达大学）；Singapore University of Technology and Design（新加坡科技设计大学）；The University of Edinburgh（爱丁堡大学）
Human identification is a key requirement for many applications in everyday life, such as personalized services, automatic surveillance, continuous authentication, and contact tracing during pandemics, etc. This work studies the problem of cross-modal human re-identification (ReID), in response to the regular human movements across camera-allowed regions (e.g., streets) and camera-restricted regions (e.g., offices) deployed with heterogeneous sensors. By leveraging the emerging low-cost RGB-D cameras and mmWave radars, we propose the first-of-its-kind vision-RF system for cross-modal multi-per…
mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual Reality
Southeast University（东南大学）；City University of Hong Kong（香港城市大学）；Michigan State University（密歇根州立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Minnesota（明尼苏达大学）
Zhejiang University（浙江大学）；Key Laboratory of Collaborative Sensing and Autonomous Unmanned Systems of Zhejiang Province（浙江省协同感知与自主无人系统重点实验室）
Gait recognition is widely used in diversified practical applications. Currently, the most prevalent approach is to recognize human gait from RGB images, owing to the progress of computer vision technologies. Nevertheless, the perception capability of RGB cameras deteriorates in rough circumstances, and visual surveillance may cause privacy invasion. Due to the robustness and non-invasive feature of millimeter wave (mmWave) radar, radar-based gait recognition has attracted increasing attention in recent years. In this research, we propose a Hierarchical Dynamic Network (HDNet) for gait recogni…
HybridSim: A Physics-Learning Hybrid Digital Twin for mmWave Human Sensing
Xiamen University Malaysia（厦门大学马来西亚分校）；The Hong Kong University of Science and Technology（香港科技大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension
Institute of Deep Perception Technology, JITRI（JITRI深度感知技术研究所）；University of Liverpool（利物浦大学）；University of Southampton（南安普顿大学）；Momoni AI（Momoni人工智能公司）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Thrust of Artificial Intelligence, HKUST (GZ)（香港科技大学（广州）人工智能学域）；Thrust of Intelligent Transportation, HKUST (GZ)（香港科技大学（广州）智能交通学域）
Michigan State University（密歇根州立大学）；Ford Motor Company（福特汽车公司）
A distinctive feature of Doppler radar is the measurement of velocity in the radial direction for radar points. However, the missing tangential velocity component hampers object velocity estimation as well as temporal integration of radar sweeps in dynamic scenes. Recognizing that fusing camera with radar provides complementary information to radar, in this paper we present a closed-form solution for the point-wise, full-velocity estimate of Doppler returns using the corresponding optical flow from camera images. Additionally, we address the association problem between radar returns and camera…
