IEEE Journal of Selected Topics in Signal Processing Special Issue on Recent Advances in Automotive Radar Signal Processing. arXiv admin note: text overlap with arXiv:2003.01816
Various autonomous or assisted driving strategies have been facilitated through the accurate and reliable perception of the environment around a vehicle. Among the commonly used sensors, radar has usually been considered as a robust and cost-effective solution even in adverse driving scenarios, e.g., weak/strong lighting or bad weather. Instead of considering to fuse the unreliable information from all available sensors, perception from pure radar data becomes a valuable alternative that is worth exploring. In this paper, we propose a deep radar object detection network, named RODNet, which is cross-supervised by a camera-radar fused algorithm without laborious annotation efforts, to effectively detect objects from the radio frequency (RF) images in real-time. First, the raw signals captured by millimeter-wave radars are transformed to RF images in range-azimuth coordinates. Second, our proposed RODNet takes a sequence of RF images as the input to predict the likelihood of objects in the radar field of view (FoV). Two customized modules are also added to handle multi-chirp information and object relative motion. Instead of using human-labeled ground truth for training, the proposed RODNet is cross-supervised by a novel 3D localization of detected objects using a camera-radar fusion (CRF) strategy in the training stage. Finally, we propose a method to evaluate the object detection performance of the RODNet. Due to no existing public dataset available for our task, we create a new dataset, named CRUW, which contains synchronized RGB and RF image sequences in various driving scenarios. With intensive experiments, our proposed cross-supervised RODNet achieves 86% average precision and 88% average recall of object detection performance, which shows the robustness to noisy scenarios in various driving conditions.
University of Washington（华盛顿大学）；Silkwave Holdings Limited（Silkwave控股有限公司）
Radar is usually more robust than the camera in severe driving scenarios, e.g., weak/strong lighting and bad weather. However, unlike RGB images captured by a camera, the semantic information from the radar signals is noticeably difficult to extract. In this paper, we propose a deep radar object detection network (RODNet), to effectively detect objects purely from the carefully processed radar frequency data in the format of range-azimuth frequency heatmaps (RAMaps). Three different 3D autoencoder based architectures are introduced to predict object confidence distribution from each snippet of…
Multi-view radar-camera fused 3D object detection provides a farther detection range and more helpful features for autonomous driving, especially under adverse weather. The current radar-camera fusion methods deliver kinds of designs to fuse radar information with camera data. However, these fusion approaches usually adopt the straightforward concatenation operation between multi-modal features, which ignores the semantic alignment with radar features and sufficient correlations across modals. In this paper, we present MVFusion, a novel Multi-View radar-camera Fusion method to achieve semantic…
TU Dortmund（多特蒙德工业大学）；HELLA Aglaia Mobile Vision GmbH（海拉阿格莱亚移动视觉有限公司）
Environment modeling utilizing sensor data fusion and object tracking is crucial for safe automated driving. In recent years, the classical occupancy grid map approach, which assumes a static environment, has been extended to dynamic occupancy grid maps, which maintain the possibility of a low-level data fusion while also estimating the position and velocity distribution of the dynamic local environment. This paper presents the further development of a previous approach. To the best of the author's knowledge, there is no publication about dynamic occupancy grid mapping with subsequent analysis…
This paper presents two variations of architecture referred to as RANet and BIRANet. The proposed architecture aims to use radar signal data along with RGB camera images to form a robust detection network that works efficiently, even in variable lighting and weather conditions such as rain, dust, fog, and others. First, radar information is fused in the feature extractor network. Second, radar points are used to generate guided anchors. Third, a method is proposed to improve region proposal network targets. BIRANet yields 72.3/75.3% average AP/AR on the NuScenes dataset, which is better than t…
Weather-Conditioned Branch Routing for Robust LiDAR-Radar 3D Object Detection
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computer and Data Science, Fuzhou University（福州大学计算机与数据科学学院）；Institute of Information Engineering, CAS（中国科学院信息工程研究所）
Enhanced 3D Object Detection via Diverse Feature Representations of 4D Radar Tensor
Graduate School of Advanced Security Science and Technology, Korea Advanced Institute of Science and Technology（韩国科学技术院高级安全科学与技术研究生院）；CCS Graduate School of Mobility, Korea Advanced Institute of Science and Technology（韩国科学技术院移动技术研究生院）；Centre Inria d’Univeristé Côte d’Azur（尼斯大学Inria中心）
physfusion: A Transformer-based Dual-Stream Radar and Vision Fusion Framework for Open Water Surface Object Detection
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces
University of Liverpool（利物浦大学）；Wuhan University of Technology（武汉理工大学）；Nantong University（南通大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
An autonomous system's perception engine must provide an accurate understanding of the environment for it to make decisions. Deep learning based object detection networks experience degradation in the performance and robustness for small and far away objects due to a reduction in object's feature map as we move to higher layers of the network. In this work, we propose a novel radar-guided spatial attention for RGB images to improve the perception quality of autonomous vehicles operating in a dynamic environment. In particular, our method improves the perception of small and long range objects,…
Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review
University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Institute of Deep Perception Technology, JITRI（江苏省产业技术研究院深度感知技术研究所）；XJTLU-JITRI Academy of Industrial Technology（西交利物浦大学-江苏省产业技术研究院工业技术学院）
FZI Research Center for Information Technology（FZI信息技术研究中心）；KIT – University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association（卡尔斯鲁厄理工学院 – 巴登-符腾堡州大学及亥姆霍兹联合会国家研究中心）
One of the most important parts of environment perception is the detection of obstacles in the surrounding of the vehicle. To achieve that, several sensors like radars, LiDARs and cameras are installed in autonomous vehicles. The produced sensor data is fused to a general representation of the surrounding. In this thesis the dynamic occupancy grid map approach of Nuss et al. is used while three goals are achieved. First, the approach of Nuss et al. to distinguish between moving and non-moving obstacles is improved by using Fully Convolutional Neural Networks to create a class prediction for ea…
Due to the trending need of building autonomous robotic perception system, sensor fusion has attracted a lot of attention amongst researchers and engineers to make best use of cross-modality information. However, in order to build a robotic platform at scale we need to emphasize on autonomous robot platform bring-up cost as well. Cameras and radars, which inherently includes complementary perception information, has potential for developing autonomous robotic platform at scale. However, there is a limited work around radar fused with Vision, compared to LiDAR fused with vision work. In this pa…
Michigan State University（密歇根州立大学）；Ford Motor Company（福特汽车公司）
While radar and video data can be readily fused at the detection level, fusing them at the pixel level is potentially more beneficial. This is also more challenging in part due to the sparsity of radar, but also because automotive radar beams are much wider than a typical pixel combined with a large baseline between camera and radar, which results in poor association between radar pixels and color pixel. A consequence is that depth completion methods designed for LiDAR and video fare poorly for radar and video. Here we propose a radar-to-pixel association stage which learns a mapping from rada…
RadarXFormer: Robust Object Detection via Cross-Dimension Fusion of 4D Radar Spectra and Images for Autonomous Driving
Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来技术全球研究院）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（中国教育部系统控制与信息处理重点实验室）；SAIC GM Wuling Automobile Company Co., Ltd.（上汽通用五菱汽车有限公司）；Guangxi Laboratory of New Energy Automobile（广西新能源汽车实验室）
Radars and cameras are mature, cost-effective, and robust sensors and have been widely used in the perception stack of mass-produced autonomous driving systems. Due to their complementary properties, outputs from radar detection (radar pins) and camera perception (2D bounding boxes) are usually fused to generate the best perception results. The key to successful radar-camera fusion is the accurate data association. The challenges in the radar-camera association can be attributed to the complexity of driving scenes, the noisy and sparse nature of radar measurements, and the depth ambiguity from…
MLF-4DRCNet: Multi-Level Fusion with 4D Radar and Camera for 3D Object Detection in Autonomous Driving
MoE Key Laboratory of Brain-Inspired Intelligence Perception and Cognition, University of Science and Technology of China（脑启发智能感知与认知教育部重点实验室，中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究院，合肥国家科学中心）；Department of Electronic Engineering and Information Science, University of Science and Technology of China（电子工程与信息科学系，中国科学技术大学）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德克萨斯大学）
LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学 cyber 系统与控制研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Technical University of Munich（慕尼黑工业大学）
SCKD: Semi-Supervised Cross-Modality Knowledge Distillation for 4D Radar Object Detection
Zhejiang University（浙江大学）；Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing（浙江省多模态通信网络与智能信息处理重点实验室）；China North Artificial Intelligence & Innovation Research Institude（北方人工智能创新研究院）
MetaOcc: Spatio-Temporal Fusion of Surround-View 4D Radar and Camera for 3D Occupancy Prediction with Dual Training Strategies
School of Automotive Studies, Tongji University（同济大学汽车学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Automobile, Chang'an University（长安大学汽车学院）；School of Information and Electrical Engineering, Hangzhou City University（杭州城市学院信息与电气工程学院）；College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）
CLAIRE: A Dual Encoder Network with RIFT Loss and Phi-3 Small Language Model Based Interpretability for Cross-Modality Synthetic Aperture Radar and Optical Land Cover Segmentation
Department of Computer Science and Engineering, United International University（计算机科学与工程系，国际大学）；Faculty of Science and Information Technology, Charles Darwin University（科学与信息技术学院，查尔斯·达尔文大学）；School of Engineering and Energy , Murdoch University（工程与能源学院，默多尼大学）；Faculty of Science and Technology, Charles Darwin University（科学与技术学院，查尔斯·达尔文大学）
Beijing University of Chemical Technology（北京化工大学）；Sugon Information Industry Co., Ltd（曙光信息产业有限公司）
The land cover classification has played an important role in remote sensing because it can intelligently identify things in one huge remote sensing image to reduce the work of humans. However, a lot of classification methods are designed based on the pixel feature or limited spatial feature of the remote sensing image, which limits the classification accuracy and universality of their methods. This paper proposed a novel method to take into the information of remote sensing image, i.e., geographic latitude-longitude information. In addition, a dual-branch convolutional neural network (CNN) cl…
State Key Laboratory of Complex Electromagnetic Environment Effects on Electronics and Information System, National University of Defense Technology（国防科技大学复杂电磁环境效应国家重点实验室）
Aircraft detection in Synthetic Aperture Radar (SAR) imagery is a challenging task in SAR Automatic Target Recognition (SAR ATR) areas due to aircraft's extremely discrete appearance, obvious intraclass variation, small size and serious background's interference. In this paper, a single-shot detector namely Attentional Feature Refinement and Alignment Network (AFRAN) is proposed for detecting aircraft in SAR images with competitive accuracy and speed. Specifically, three significant components including Attention Feature Fusion Module (AFFM), Deformable Lateral Connection Module (DLCM) and Anc…
University of Science and Technology of China（中国科学技术大学）
Ground Penetrating Radar (GPR) has been widely used to estimate the healthy operation of some urban roads and underground facilities. When identifying subsurface anomalies by GPR in an area, the obtained data could be unbalanced, and the numbers and types of possible underground anomalies could not be acknowledged in advance. In this paper, a novel method is proposed to improve the subsurface anomaly detection from GPR B-scan images. A normal (i.e. without subsurface objects) GPR image section is firstly collected in the detected area. Concerning that the GPR image is essentially the represent…
Cloud Removal With PolSAR-Optical Data Fusion Using A Two-Flow Residual Network
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；International Research Center of Big Data for Sustainable Development Goals（国际可持续发展大数据研究中心）；College of Resources and Environment, University of Chinese Academy of Sciences（中国科学院大学资源与环境学院）
Heterogeneous SAR-optical fusion for near-real-time land use and land cover mapping under cloud contamination: A novel framework and global benchmark dataset
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing（信息工程测绘与遥感国家重点实验室）；School of Computer Science and Information Engineering（计算机科学与信息工程学院）；Hangzhou International Innovation Institute（杭州国际创新研究院）；Oriental Space Port Research Institute（东方航天港研究院）；Hubei Luojia Laboratory（湖北珞珈实验室）
