3D occupancy-based perception pipeline has significantly advanced autonomous driving by capturing detailed scene descriptions and demonstrating strong generalizability across various object categories and shapes. Current methods predominantly rely on LiDAR or camera inputs for 3D occupancy prediction. These methods are susceptible to adverse weather conditions, limiting the all-weather deployment of self-driving cars. To improve perception robustness, we leverage the recent advances in automotive radars and introduce a novel approach that utilizes 4D imaging radar sensors for 3D occupancy prediction. Our method, RadarOcc, circumvents the limitations of sparse radar point clouds by directly processing the 4D radar tensor, thus preserving essential scene details. RadarOcc innovatively addresses the challenges associated with the voluminous and noisy 4D radar data by employing Doppler bins descriptors, sidelobe-aware spatial sparsification, and range-wise self-attention mechanisms. To minimize the interpolation errors associated with direct coordinate transformations, we also devise a spherical-based feature encoding followed by spherical-to-Cartesian feature aggregation. We benchmark various baseline methods based on distinct modalities on the public K-Radar dataset. The results demonstrate RadarOcc's state-of-the-art performance in radar-based 3D occupancy prediction and promising results even when compared with LiDAR- or camera-based methods. Additionally, we present qualitative evidence of the superior performance of 4D radar in adverse weather conditions and explore the impact of key pipeline components through ablation studies.
REOcc: Camera-Radar Fusion with Radar Feature Enrichment for 3D Occupancy Prediction
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院电子工程学院);Department of Automobile and IT Convergence, Kookmin University(韩国庆熙大学汽车与IT融合系);Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院移动技术研究生院);Robotics Program, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院机器人项目)
LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera
Institute of Cyber-Systems and Control, Zhejiang University(浙江大学 cyber 系统与控制研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Technical University of Munich(慕尼黑工业大学)
MetaOcc: Spatio-Temporal Fusion of Surround-View 4D Radar and Camera for 3D Occupancy Prediction with Dual Training Strategies
School of Automotive Studies, Tongji University(同济大学汽车学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Automobile, Chang'an University(长安大学汽车学院);School of Information and Electrical Engineering, Hangzhou City University(杭州城市学院信息与电气工程学院);College of Science and Engineering, James Cook University(詹姆斯库克大学科学与工程学院)
Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception
School of Automotive Studies, Tongji University(同济大学汽车学院);Momoni AI;Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);Chair of Automotive Engineering, Technische Universität Berlin(柏林技术大学汽车工程学系);College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);School of Information and Electrical Engineering, Hangzhou City University(杭州城市学院信息与电气工程学院)
CRAB: Camera-Radar Fusion for Reducing Depth Ambiguity in Backward Projection based View Transformation
Interdisciplinary Program in Artificial intelligence, Seoul National University(人工智能跨学科项目,首尔国立大学);Cho Chun Shik Graduate School of Mobility, KAIST(移动性研究生院,韩国科学技术院);dot Inc(42dot公司);Electronics and Telecommunications Research Institute (ETRI)(电子电信研究院);Department of Automobile and IT Convergence, Kookmin University(汽车与IT融合系,韩国釜山大学)
LiDAR-BIND-T: Improved and Temporally Consistent Sensor Modality Translation and Fusion for Robotic Applications
IDLab - Faculty of Applied Engineering, University of Antwerp - imec(IDLab - 应用工程学院,安特卫普大学 - imec);Cosys-Lab - Faculty of Applied Engineering, University of Antwerp(Cosys-Lab - 应用工程学院,安特卫普大学);Flanders Make Strategic Research Centre(弗拉芒制造战略研究中心)
本文提出 LiDAR-BIND-T,通过时间嵌入相似度、运动对齐损失和窗口化时间融合增强异构传感器到 LiDAR 的转换,并以新指标验证其提升 SLAM 稳定性与精度。
OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving
School of Automotive Studies, Tongji University(同济大学汽车学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);AI Foundation(2077AI基金会);Momoni AI;College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);School of Information and Electrical Engineering, Hangzhou City University(杭州城市学院信息与电气工程学院)
OG-PCL: Efficient Sparse Point Cloud Processing for Human Activity Recognition
Key Laboratory of Underwater Acoustic Signal Processing of Ministry of Education, Southeast University, Nanjing, 211189, China(教育部水下声学信号处理重点实验室,东南大学,南京)