The perception of autonomous vehicles using radars has attracted increased research interest due its ability to operate in fog and bad weather. However, training radar models is hindered by the cost and difficulty of annotating large-scale radar data. To overcome this bottleneck, we propose a self-supervised learning framework to leverage the large amount of unlabeled radar data to pre-train radar-only embeddings for self-driving perception tasks. The proposed method combines radar-to-radar and radar-to-vision contrastive losses to learn a general representation from unlabeled radar heatmaps paired with their corresponding camera images. When used for downstream object detection, we demonstrate that the proposed self-supervision framework can improve the accuracy of state-of-the-art supervised baselines by $5.8\%$ in mAP. Code is available at \url{https://github.com/yiduohao/Radical}.
Brandeis University（布兰迪斯大学）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
We consider the object recognition problem in autonomous driving using automotive radar sensors. Comparing to Lidar sensors, radar is cost-effective and robust in all-weather conditions for perception in autonomous driving. However, radar signals suffer from low angular resolution and precision in recognizing surrounding objects. To enhance the capacity of automotive radar, in this work, we exploit the temporal information from successive ego-centric bird-eye-view radar image frames for radar object recognition. We leverage the consistency of an object's existence and attributes (size, orienta…
Beijing University of Posts and Telecommunications（北京邮电大学）
With autonomous driving developing in a booming stage, accurate object detection in complex scenarios attract wide attention to ensure the safety of autonomous driving. Millimeter wave (mmWave) radar and vision fusion is a mainstream solution for accurate obstacle detection. This article presents a detailed survey on mmWave radar and vision fusion based obstacle detection methods. First, we introduce the tasks, evaluation criteria, and datasets of object detection for autonomous driving. The process of mmWave radar and vision fusion is then divided into three parts: sensor deployment, sensor c…
RadarXFormer: Robust Object Detection via Cross-Dimension Fusion of 4D Radar Spectra and Images for Autonomous Driving
Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来技术全球研究院）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（中国教育部系统控制与信息处理重点实验室）；SAIC GM Wuling Automobile Company Co., Ltd.（上汽通用五菱汽车有限公司）；Guangxi Laboratory of New Energy Automobile（广西新能源汽车实验室）
A Survey of Deep Learning Based Radar and Vision Fusion for 3D Object Detection in Autonomous Driving
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Electronic and Information Engineering, Suzhou University of Science and Technology（苏州科技大学电子与信息工程学院）
The next-generation high-resolution automotive radar (4D radar) can provide additional elevation measurement and denser point clouds, which has great potential for 3D sensing in autonomous driving. In this paper, we introduce a dataset named TJ4DRadSet with 4D radar points for autonomous driving research. The dataset was collected in various driving scenarios, with a total of 7757 synchronized frames in 44 consecutive sequences, which are well annotated with 3D bounding boxes and track ids. We provide a 4D radar-based 3D object detection baseline for our dataset to demonstrate the effectivenes…
ZFusion: An Effective Fuser of Camera and 4D Radar for 3D Object Perception in Autonomous Driving
School of Data Science, Fudan University（复旦大学数据科学学院）；ZF (China) Investment Co., Ltd.（采埃孚（中国）投资有限公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Radars and cameras are mature, cost-effective, and robust sensors and have been widely used in the perception stack of mass-produced autonomous driving systems. Due to their complementary properties, outputs from radar detection (radar pins) and camera perception (2D bounding boxes) are usually fused to generate the best perception results. The key to successful radar-camera fusion is the accurate data association. The challenges in the radar-camera association can be attributed to the complexity of driving scenes, the noisy and sparse nature of radar measurements, and the depth ambiguity from…
Timely Fusion of Surround Radar/Lidar for Object Detection in Autonomous Driving Systems
City University of Hong Kong（香港城市大学）；Yale University（耶鲁大学）；The Chinese University of Hong Kong（香港中文大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Mask-RadarNet: Enhancing Transformer With Spatial-Temporal Semantic Context for Radar Object Detection in Autonomous Driving
University of Science and Technology of China（中国科学技术大学）；Wuhan Electronic Information Institute（武汉电子信息研究所）；Università degli Studi “Niccolò Cusano”（尼科洛·库萨诺大学）
MLF-4DRCNet: Multi-Level Fusion with 4D Radar and Camera for 3D Object Detection in Autonomous Driving
MoE Key Laboratory of Brain-Inspired Intelligence Perception and Cognition, University of Science and Technology of China（脑启发智能感知与认知教育部重点实验室，中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究院，合肥国家科学中心）；Department of Electronic Engineering and Information Science, University of Science and Technology of China（电子工程与信息科学系，中国科学技术大学）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德克萨斯大学）
Detecting obstacles is crucial for safe and efficient autonomous driving. To this end, we present NVRadarNet, a deep neural network (DNN) that detects dynamic obstacles and drivable free space using automotive RADAR sensors. The network utilizes temporally accumulated data from multiple RADAR sensors to detect dynamic obstacles and compute their orientation in a top-down bird's-eye view (BEV). The network also regresses drivable free space to detect unclassified obstacles. Our DNN is the first of its kind to utilize sparse RADAR signals in order to perform obstacle and free space detection in …
Exploring Radar Data Representations in Autonomous Driving: A Comprehensive Review
School of Information Engineering, Yancheng Institute Technology（盐城理工学院信息工程学院）；Thrust of Artificial Intelligence and Thrust of Intelligent Transportation, The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）人工智能与智能交通研究所）；Faculty of Science and Engineering, University of Liverpool（利物浦大学科学与工程学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学高级技术学院）；School of Information Science and Technology, Nantong University（南通大学信息科学与技术学院）
CORENet: Cross-Modal 4D Radar Denoising Network with LiDAR Supervision for Autonomous Driving
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Control Engineering（北京控制工程研究所）
Robert Bosch GmbH（罗伯特·博世股份有限公司）；University of Stuttgart（斯图加特大学）
Simulation-based testing is a promising approach to significantly reduce the validation effort of automated driving functions. Realistic models of environment perception sensors such as camera, radar and lidar play a key role in this testing strategy. A generally accepted method to validate these sensor models does not yet exist. Particularly radar has traditionally been one of the most difficult sensors to model. Although promising as an alternative to real test drives, virtual tests are time-consuming due to the fact that they simulate the entire radar system in detail, using computation-int…
4D-CAAL: 4D Radar-Camera Calibration and Auto-Labeling for Autonomous Driving
School of Information Engineering, Yancheng Institute of Technology（信息工程学院，盐城职业技术学院）；School of Navigation, Wuhan University of Technology（导航学院，武汉理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；School of Information Engineering, Yancheng Institute Technology（信息工程学院，盐城职业技术学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（先进技术学院，西安交通大学利物浦大学）；School of Information Science and Technology, Nantong University（信息科学与技术学院，南通大学）；State Key Laboratory of Maritime Technology and Safety（船舶技术与安全国家重点实验室）
Robert Bosch GmbH（罗伯特·博世有限公司）；University of Stuttgart（斯图加特大学）
The usage of environment sensor models for virtual testing is a promising approach to reduce the testing effort of autonomous driving. However, in order to deduce any statements regarding the performance of an autonomous driving function based on simulation, the sensor model has to be validated to determine the discrepancy between the synthetic and real sensor data. Since a certain degree of divergence can be assumed to exist, the sufficient level of fidelity must be determined, which poses a major challenge. In particular, a method for quantifying the fidelity of a sensor model does not exist…
USVTrack: USV-Based 4D Radar-Camera Tracking Dataset for Autonomous Driving in Inland Waterways
School of Information Engineering, Yancheng Institute Technology（盐城理工学院信息工程学院）；Thrust of Artificial Intelligence and Thrust of Intelligent Transportation, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）人工智能与智能交通研究部）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学先进科技学院）；School of Navigation, Wuhan University of Technology（武汉理工大学航海学院）
CLLAP: Contrastive Learning-based LiDAR-Augmented Pretraining for Enhanced Radar-Camera Fusion
Wuhan University of Technology（武汉理工大学）；University of North Carolina at Charlotte（北卡罗来纳州立大学）；Wuhan University（武汉大学）；Southwest Jiaotong University（西南交通大学）；The Hong Kong Polytechnic University（香港理工大学）
ELMAR: Enhancing LiDAR Detection with 4D Radar Motion Awareness and Cross-modal Uncertainty
Infineon Technologies AG and the Technical University of Munich（英飞凌科技有限公司和技术大学慕尼黑）；China University of Geosciences（中国地质大学）；Technical University of Munich（技术大学慕尼黑）
Uber Advanced Technologies Group（Uber先进技术集团）；University of Waterloo（滑铁卢大学）
We tackle the problem of exploiting Radar for perception in the context of self-driving as Radar provides complementary information to other sensors such as LiDAR or cameras in the form of Doppler velocity. The main challenges of using Radar are the noise and measurement ambiguities which have been a struggle for existing simple input or output fusion methods. To better address this, we propose a new solution that exploits both LiDAR and Radar sensors for perception. Our approach, dubbed RadarNet, features a voxel-based early fusion and an attention-based late fusion, which learn from data to …
BMW of North America（北美宝马公司）；Graz University of Technology（格拉茨工业大学）
Collecting realistic driving trajectories is crucial for training machine learning models that imitate human driving behavior. Most of today's autonomous driving datasets contain only a few trajectories per location and are recorded with test vehicles that are cautiously driven by trained drivers. In particular in interactive scenarios such as highway merges, the test driver's behavior significantly influences other vehicles. This influence prevents recording the whole traffic space of human driving behavior. In this work, we present a novel methodology to extract trajectories of traffic objec…
RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）
Robert Bosch GmbH（罗伯特·博世有限公司）；University of Stuttgart（斯图加特大学）
With the increasing safety validation requirements for the release of a self-driving car, alternative approaches, such as simulation-based testing, are emerging in addition to conventional real-world testing. In order to rely on virtual tests the employed sensor models have to be validated. For this reason, it is necessary to quantify the discrepancy between simulation and reality in order to determine whether a certain fidelity is sufficient for a desired intended use. There exists no sound method to measure this simulation-to-reality gap of radar perception for autonomous driving. We address…
A complete overview of the surrounding vehicle environment is important for driver assistance systems and highly autonomous driving. Fusing results of multiple sensor types like camera, radar and lidar is crucial for increasing the robustness. The detection and classification of objects like cars, bicycles or pedestrians has been analyzed in the past for many sensor types. Beyond that, it is also helpful to refine these classes and distinguish for example between different pedestrian types or activities. This task is usually performed on camera data, though recent developments are based on rad…
MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Driving
Department of Electronic Engineering and Information Science, University of Science and Technology of China（电子工程与信息科学系，中国科学技术大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所，东部技术研究所）
Advancing Autonomous Driving: DepthSense with Radar and Spatial Attention
School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology（全州科学技术院电气工程与计算机科学学院）；Division of National Science and Technology Data (Large Scale AI Research Group), Korea Institute of Science and Technology Information (KISTI)（国家科学技术数据 division（大规模人工智能研究组），韩国科学技术信息研究院）；Department of Information Systems, College of Computer Sciences and Information Technology, King Faisal University（国王法尔萨大学信息系统系，计算机科学与信息科技学院）
Cross-Dataset Experimental Study of Radar-Camera Fusion in Bird's-Eye View
Stellantis, Opel Automobile GmbH（斯泰兰蒂斯集团，欧宝汽车有限公司）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau（莱茵兰-普法尔茨凯泽斯劳滕-兰道工业大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
School of Electrical Engineering and Computer Sciences, University of Ottawa（渥太华大学电气工程与计算机科学学院）；Sensorcortek Inc（森索科泰克公司）；Valeo（法雷奥公司）
Camera and Lidar processing have been revolutionized with the rapid development of deep learning model architectures. Automotive radar is one of the crucial elements of automated driver assistance and autonomous driving systems. Radar still relies on traditional signal processing techniques, unlike camera and Lidar based methods. We believe this is the missing link to achieve the most robust perception system. Identifying drivable space and occupied space is the first step in any autonomous decision making task. Occupancy grid map representation of the environment is often used for this purpos…
Environmental perception with the multi-modal fusion of radar and camera is crucial in autonomous driving to increase accuracy, completeness, and robustness. This paper focuses on utilizing millimeter-wave (MMW) radar and camera sensor fusion for 3D object detection. A novel method that realizes the feature-level fusion under the bird's-eye view (BEV) for a better feature representation is proposed. Firstly, radar points are augmented with temporal accumulation and sent to a spatial-temporal encoder for radar feature extraction. Meanwhile, multi-scale image 2D features which adapt to various s…
Simulating Automotive Radar with Lidar and Camera Inputs
Institute of Robotics and Automatic Information System, Nankai University（机器人与自动信息系统研究所，南开大学）；Tianjin Key Laboratory of Intelligent Robotics（天津智能机器人重点实验室）；TBI center, Nankai University（南开大学TBI中心）；Department of Robotics, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)（人工智能 Mohamed Bin Zayed 大学机器人系）
