Due to the trending need of building autonomous robotic perception system, sensor fusion has attracted a lot of attention amongst researchers and engineers to make best use of cross-modality information. However, in order to build a robotic platform at scale we need to emphasize on autonomous robot platform bring-up cost as well. Cameras and radars, which inherently includes complementary perception information, has potential for developing autonomous robotic platform at scale. However, there is a limited work around radar fused with Vision, compared to LiDAR fused with vision work. In this paper, we tackle this gap with a survey on Vision-Radar fusion approaches for a BEV object detection system. First we go through the background information viz., object detection tasks, choice of sensors, sensor setup, benchmark datasets and evaluation metrics for a robotic perception system. Later, we cover per-modality (Camera and RADAR) data representation, then we go into detail about sensor fusion techniques based on sub-groups viz., early-fusion, deep-fusion, and late-fusion to easily understand the pros and cons of each method. Finally, we propose possible future trends for vision-radar fusion to enlighten future research. Regularly updated summary can be found at: https://github.com/ApoorvRoboticist/Vision-RADAR-Fusion-BEV-Survey
RC-BEVFusion: A Plug-In Module for Radar-Camera Bird's Eye View Feature Fusion
Stellantis, Opel Automobile GmbH（斯泰兰蒂斯集团欧宝汽车有限公司）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau（莱茵兰-普法尔茨凯泽斯劳滕-兰道工业大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
Stanford University（斯坦福大学）；Carnegie Mellon University（卡内基梅隆大学）；Toyota Research Institute（丰田研究院）
Building 3D perception systems for autonomous vehicles that do not rely on high-density LiDAR is a critical research problem because of the expense of LiDAR systems compared to cameras and other sensors. Recent research has developed a variety of camera-only methods, where features are differentiably "lifted" from the multi-camera images onto the 2D ground plane, yielding a "bird's eye view" (BEV) feature representation of the 3D space around the vehicle. This line of work has produced a variety of novel "lifting" methods, but we observe that other details in the training setups have shifted a…
RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）
CRAB: Camera-Radar Fusion for Reducing Depth Ambiguity in Backward Projection based View Transformation
Interdisciplinary Program in Artificial intelligence, Seoul National University（人工智能跨学科项目，首尔国立大学）；Cho Chun Shik Graduate School of Mobility, KAIST（移动性研究生院，韩国科学技术院）；dot Inc（42dot公司）；Electronics and Telecommunications Research Institute (ETRI)（电子电信研究院）；Department of Automobile and IT Convergence, Kookmin University（汽车与IT融合系，韩国釜山大学）
Environmental perception with the multi-modal fusion of radar and camera is crucial in autonomous driving to increase accuracy, completeness, and robustness. This paper focuses on utilizing millimeter-wave (MMW) radar and camera sensor fusion for 3D object detection. A novel method that realizes the feature-level fusion under the bird's-eye view (BEV) for a better feature representation is proposed. Firstly, radar points are augmented with temporal accumulation and sent to a spatial-temporal encoder for radar feature extraction. Meanwhile, multi-scale image 2D features which adapt to various s…
SFGFusion: Surface Fitting Guided 3D Object Detection with 4D Radar and Camera Fusion
Radar Technology Research Institute, School of Information and Electronics, Beijing Institute of Technology（雷达技术研究院，信息电子学院，北京理工大学）；School of Computer Science and Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）；Innovative Equipment Research Institute, Beijing Institute of Technology（创新装备研究院，北京理工大学）；Key Laboratory of Electronic and Information Technology in Satellite Navigation (Beijing Institute of Technology), Ministry of Education（卫星导航电子信息技术重点实验室（北京理工大学），教育部）；Beijing Racobit Electronic Information Technology Co., Ltd.（北京瑞科比特电子信息技术有限公司）
Currently, detecting 3D objects in Bird's-Eye-View (BEV) is superior to other 3D detectors for autonomous driving and robotics. However, transforming image features into BEV necessitates special operators to conduct feature sampling. These operators are not supported on many edge devices, bringing extra obstacles when deploying detectors. To address this problem, we revisit the generation of BEV representation and propose detecting objects in perspective BEV -- a new BEV representation that does not require feature sampling. We demonstrate that perspective BEV features can likewise enjoy the b…
Hamilton Institute Maynooth University（汉密尔顿研究所梅努斯大学）；Valeo Vision Systems（法雷奥视觉系统公司）；Maynooth University（梅努斯大学）
Generating a detailed near-field perceptual model of the environment is an important and challenging problem in both self-driving vehicles and autonomous mobile robotics. A Bird Eye View (BEV) map, providing a panoptic representation, is a commonly used approach that provides a simplified 2D representation of the vehicle surroundings with accurate semantic level segmentation for many downstream tasks. Current state-of-the art approaches to generate BEV-maps employ a Convolutional Neural Network (CNN) backbone to create feature-maps which are passed through a spatial transformer to project the …
Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception
School of Automotive Studies, Tongji University（同济大学汽车学院）；Momoni AI；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；Chair of Automotive Engineering, Technische Universität Berlin（柏林技术大学汽车工程学系）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；School of Information and Electrical Engineering, Hangzhou City University（杭州城市学院信息与电气工程学院）
OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird's-eye-view Vehicle Semantic Segmentation
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 Mobility学院）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（清华大学智能绿色车辆与交通全国重点实验室）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）
A Survey of Deep Learning Based Radar and Vision Fusion for 3D Object Detection in Autonomous Driving
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Electronic and Information Engineering, Suzhou University of Science and Technology（苏州科技大学电子与信息工程学院）
BEV$^2$PR: BEV-Enhanced Visual Place Recognition with Structural Cues
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Institute of Cyber-Systems and Control, Zhejiang University（浙江大学智能系统与控制研究所）
Northeastern University（东北大学）；Peking University（北京大学）；Technical University in Munich（慕尼黑工业大学）；Ulster University（阿尔斯特大学）
The progress of LiDAR-based 3D object detection has significantly enhanced developments in autonomous driving and robotics. However, due to the limitations of LiDAR sensors, object shapes suffer from deterioration in occluded and distant areas, which creates a fundamental challenge to 3D perception. Existing methods estimate specific 3D shapes and achieve remarkable performance. However, these methods rely on extensive computation and memory, causing imbalances between accuracy and real-time performance. To tackle this challenge, we propose a novel LiDAR-based 3D object detection model named B…
Detecting obstacles is crucial for safe and efficient autonomous driving. To this end, we present NVRadarNet, a deep neural network (DNN) that detects dynamic obstacles and drivable free space using automotive RADAR sensors. The network utilizes temporally accumulated data from multiple RADAR sensors to detect dynamic obstacles and compute their orientation in a top-down bird's-eye view (BEV). The network also regresses drivable free space to detect unclassified obstacles. Our DNN is the first of its kind to utilize sparse RADAR signals in order to perform obstacle and free space detection in …
BEVPose: Unveiling Scene Semantics through Pose-Guided Multi-Modal BEV Alignment
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；The University of Adelaide（阿德莱德大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion
University of Science and Technology of China（中国科学技术大学）；The University of Adelaide（阿德莱德大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Monocular visual-inertial odometry (VIO) is a critical problem in robotics and autonomous driving. Traditional methods solve this problem based on filtering or optimization. While being fully interpretable, they rely on manual interference and empirical parameter tuning. On the other hand, learning-based approaches allow for end-to-end training but require a large number of training data to learn millions of parameters. However, the non-interpretable and heavy models hinder the generalization ability. In this paper, we propose a fully differentiable, and interpretable, bird-eye-view (BEV) base…
UniBEVFusion: Unified Radar-Vision BEVFusion for 3D Object Detection
Institute of Deep Perception Technology, JITRI（捷瑞深度感知技术研究所）；University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；XJTLU-JITRI Academy of Technology, Xi’an Jiaotong-Liverpool University（西交利物浦大学-捷瑞技术学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Depth-aware Fusion Method based on Image and 4D Radar Spectrum for 3D Object Detection
The Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来全球技术研究院）；Shanghai Jiao Tong University（上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（教育部系统控制与信息处理重点实验室）
ELMAR: Enhancing LiDAR Detection with 4D Radar Motion Awareness and Cross-modal Uncertainty
Infineon Technologies AG and the Technical University of Munich（英飞凌科技有限公司和技术大学慕尼黑）；China University of Geosciences（中国地质大学）；Technical University of Munich（技术大学慕尼黑）
LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学 cyber 系统与控制研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Technical University of Munich（慕尼黑工业大学）
