Simultaneous Localization and Mapping (SLAM) is moving towards a robust perception age. However, LiDAR- and visual- SLAM may easily fail in adverse conditions (rain, snow, smoke and fog, etc.). In comparison, SLAM based on 4D Radar, thermal camera and IMU can work robustly. But only a few literature can be found. A major reason is the lack of related datasets, which seriously hinders the research. Even though some datasets are proposed based on 4D radar in past four years, they are mainly designed for object detection, rather than SLAM. Furthermore, they normally do not include thermal camera. Therefore, in this paper, NTU4DRadLM is presented to meet this requirement. The main characteristics are: 1) It is the only dataset that simultaneously includes all 6 sensors: 4D radar, thermal camera, IMU, 3D LiDAR, visual camera and RTK GPS. 2) Specifically designed for SLAM tasks, which provides fine-tuned ground truth odometry and intentionally formulated loop closures. 3) Considered both low-speed robot platform and fast-speed unmanned vehicle platform. 4) Covered structured, unstructured and semi-structured environments. 5) Considered both middle- and large- scale outdoor environments, i.e., the 6 trajectories range from 246m to 6.95km. 6) Comprehensively evaluated three types of SLAM algorithms. Totally, the dataset is around 17.6km, 85mins, 50GB and it will be accessible from this link: https://github.com/junzhang2016/NTU4DRadLM
Seoul National University（首尔大学）；AI COLLEGE（人工智能学院）
Even though many existing 3D object detection algorithms rely mostly on camera and LiDAR, camera and LiDAR are prone to be affected by harsh weather and lighting conditions. On the other hand, radar is resistant to such conditions. However, research has found only recently to apply deep neural networks on radar data. In this paper, we introduce a deep learning approach to 3D object detection with radar only. To the best of our knowledge, we are the first ones to demonstrate a deep learning-based 3D object detection model with radar only that was trained on the public radar dataset. To overcome…
Towards Dense and Accurate Radar Perception Via Efficient Cross-Modal Diffusion Model
Huzhou Institute, Zhejiang University（浙江大学湖州研究院）；Nanyang Technological University（南洋理工大学）；School of Cyber Science and Technology, University of Science and Technology of China（中国科学技术大学网络空间安全学院）；Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院 cyber系统与控制研究所）
Multi-view radar-camera fused 3D object detection provides a farther detection range and more helpful features for autonomous driving, especially under adverse weather. The current radar-camera fusion methods deliver kinds of designs to fuse radar information with camera data. However, these fusion approaches usually adopt the straightforward concatenation operation between multi-modal features, which ignores the semantic alignment with radar features and sufficient correlations across modals. In this paper, we present MVFusion, a novel Multi-View radar-camera Fusion method to achieve semantic…
CARIAD SE；University of Bonn（波恩大学）；University of Oxford（牛津大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉玛尔机器学习与人工智能研究所）
Scene understanding is crucial for autonomous robots in dynamic environments for making future state predictions, avoiding collisions, and path planning. Camera and LiDAR perception made tremendous progress in recent years, but face limitations under adverse weather conditions. To leverage the full potential of multi-modal sensor suites, radar sensors are essential for safety critical tasks and are already installed in most new vehicles today. In this paper, we address the problem of semantic segmentation of moving objects in radar point clouds to enhance the perception of the environment with…
We propose DeepFusion, a modular multi-modal architecture to fuse lidars, cameras and radars in different combinations for 3D object detection. Specialized feature extractors take advantage of each modality and can be exchanged easily, making the approach simple and flexible. Extracted features are transformed into bird's-eye-view as a common representation for fusion. Spatial and semantic alignment is performed prior to fusing modalities in the feature space. Finally, a detection head exploits rich multi-modal features for improved 3D detection performance. Experimental results for lidar-came…
WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces
University of Liverpool（利物浦大学）；Wuhan University of Technology（武汉理工大学）；Nantong University（南通大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion
State Key Laboratory of Ocean Sensing and Ocean College, Zhejiang University（浙江大学海洋传感与海洋学院海洋传感技术国家重点实验室）；Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University（浙江大学海洋传感技术与装备教育部工程研究中心）；Donghai Laboratory（东海实验室）；Ocean College, Institute of Marine Electronics and Intelligent Systems, Zhejiang University（浙江大学海洋学院海洋电子与智能系统研究所）
M^3Detection: Multi-Frame Multi-Level Feature Fusion for Multi-Modal 3D Object Detection with Camera and 4D Imaging Radar
Radar Technology Research Institute, School of Information and Electronics, Beijing Institute of Technology（雷达技术研究所，信息电子学院，北京理工大学）；Key Laboratory of Electronic and Information Technology in Satellite Navigation, Ministry of Education（卫星导航电子信息技术重点实验室，教育部）；School of Computer Science, Beijing Institute of Technology（计算机学院，北京理工大学）；Beijing Racobit Electronic Information Technology Co., Ltd.（北京Racobit电子信息技术有限公司）
University of Washington（华盛顿大学）；Silkwave Holdings Limited（Silkwave控股有限公司）
Radar is usually more robust than the camera in severe driving scenarios, e.g., weak/strong lighting and bad weather. However, unlike RGB images captured by a camera, the semantic information from the radar signals is noticeably difficult to extract. In this paper, we propose a deep radar object detection network (RODNet), to effectively detect objects purely from the carefully processed radar frequency data in the format of range-azimuth frequency heatmaps (RAMaps). Three different 3D autoencoder based architectures are introduced to predict object confidence distribution from each snippet of…
UniBEVFusion: Unified Radar-Vision BEVFusion for 3D Object Detection
Institute of Deep Perception Technology, JITRI（捷瑞深度感知技术研究所）；University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；XJTLU-JITRI Academy of Technology, Xi’an Jiaotong-Liverpool University（西交利物浦大学-捷瑞技术学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Deep reinforcement learning (RL), where the agent learns from mistakes, has been successfully applied to a variety of tasks. With the aim of learning collision-free policies for unmanned vehicles, deep RL has been used for training with various types of data, such as colored images, depth images, and LiDAR point clouds, without the use of classic map--localize--plan approaches. However, existing methods are limited by their reliance on cameras and LiDAR devices, which have degraded sensing under adverse environmental conditions (e.g., smoky environments). In response, we propose the use of sin…
Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；State Key Laboratory of Industrial Control Technology, Zhejiang University（浙江大学工业控制技术国家重点实验室）；Hangzhou Dianzi University（杭州电子科技大学）；Pillar of Information Systems Technology and Design, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计 pillar）
Weather-Conditioned Branch Routing for Robust LiDAR-Radar 3D Object Detection
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computer and Data Science, Fuzhou University（福州大学计算机与数据科学学院）；Institute of Information Engineering, CAS（中国科学院信息工程研究所）
Edinburgh Centre for Robotics（爱丁堡机器人中心）；Heriot-Watt University（赫瑞-瓦特大学）
Numerous Simultaneous Localization and Mapping (SLAM) algorithms have been presented in last decade using different sensor modalities. However, robust SLAM in extreme weather conditions is still an open research problem. In this paper, RadarSLAM, a full radar based graph SLAM system, is proposed for reliable localization and mapping in large-scale environments. It is composed of pose tracking, local mapping, loop closure detection and pose graph optimization, enhanced by novel feature matching and probabilistic point cloud generation on radar images. Extensive experiments are conducted on a pu…
Achelous++: Power-Oriented Water-Surface Panoptic Perception Framework on Edge Devices based on Vision-Radar Fusion and Pruning of Heterogeneous Modalities
University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；JITRI（江苏省产业技术研究院）；Nantong University（南通大学）
Multispectral Detection Transformer with Infrared-Centric Feature Fusion
Artificial Intelligence Graduate School, Gwangju Institute of Science and Technology (GIST)（人工智能研究生院，全州科学技术院（GIST））；School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology (GIST)（电气工程与计算机科学学院，全州科学技术院（GIST））
Gwangju Institute of Science and Technology（光州科学技术院）
Autonomous vehicles are conceived to provide safe and secure services by validating the safety standards as indicated by SOTIF-ISO/PAS-21448 (Safety of the intended functionality). Keeping in this context, the perception of the environment plays an instrumental role in conjunction with localization, planning and control modules. As a pivotal algorithm in the perception stack, object detection provides extensive insights into the autonomous vehicle's surroundings. Camera and Lidar are extensively utilized for object detection among different sensor modalities, but these exteroceptive sensors ha…
National Yang Ming Chiao Tung University（阳明交通大学）；University of Washington（华盛顿大学）
This paper introduces a novel human pose estimation benchmark, Human Pose with Millimeter Wave Radar (HuPR), that includes synchronized vision and radio signal components. This dataset is created using cross-calibrated mmWave radar sensors and a monocular RGB camera for cross-modality training of radar-based human pose estimation. There are two advantages of using mmWave radar to perform human pose estimation. First, it is robust to dark and low-light conditions. Second, it is not visually perceivable by humans and thus, can be widely applied to applications with privacy concerns, e.g., survei…
Robust 3D object detection is critical for safe autonomous driving. Camera and radar sensors are synergistic as they capture complementary information and work well under different environmental conditions. Fusing camera and radar data is challenging, however, as each of the sensors lacks information along a perpendicular axis, that is, depth is unknown to camera and elevation is unknown to radar. We propose the camera-radar matching network CramNet, an efficient approach to fuse the sensor readings from camera and radar in a joint 3D space. To leverage radar range measurements for better came…
FIRE-LIVWO: Robust LiDAR-Inertial-Visual-Wheel Odometry via Failure-Immune mmWave Radar Enhancement
School of Mechatronic Engineering, China University of Mining and Technology（中国矿业大学机电工程学院）；Jiangsu Collaborative Innovation Center of Intelligent Mining Equipment, China University of Mining and Technology（中国矿业大学江苏省智能采矿装备协同创新中心）；National Key Laboratory of Intelligent Mining Equipment Technology, China University of Mining and Technology（中国矿业大学智能采矿装备技术全国重点实验室）
WS-DETR: Robust Water Surface Object Detection through Vision-Radar Fusion with Detection Transformer
College of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Graz University of Technology（格拉茨工业大学）；Virtual Vehicle Research（虚拟车辆研究院）
University of Edinburgh（爱丁堡大学）；Delft University of Technology（代尔夫特理工大学）
This work proposes a novel approach to 4D radar-based scene flow estimation via cross-modal learning. Our approach is motivated by the co-located sensing redundancy in modern autonomous vehicles. Such redundancy implicitly provides various forms of supervision cues to the radar scene flow estimation. Specifically, we introduce a multi-task model architecture for the identified cross-modal learning problem and propose loss functions to opportunistically engage scene flow estimation using multiple cross-modal constraints for effective model training. Extensive experiments show the state-of-the-a…
Compared to the onboard camera and laser scanner, radar sensor provides lighting and weather invariant sensing, which is naturally suitable for long-term localization under adverse conditions. However, radar data is sparse and noisy, resulting in challenges for radar mapping. On the other hand, the most popular available map currently is built by lidar. In this paper, we propose an end-to-end deep learning framework for Radar Localization on Lidar Map (RaLL) to bridge the gap, which not only achieves the robust radar localization but also exploits the mature lidar mapping technique, thus reduc…
MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency
Fraunhofer Institute IVI（弗劳恩霍夫交通系统研究所）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Cambridge（剑桥大学）；Univ. of Applied Sciences Ingolstadt（英戈尔施塔特应用科学大学）
LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学 cyber 系统与控制研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Technical University of Munich（慕尼黑工业大学）
We present a heterogeneous localization framework for solving radar global localization and pose tracking on pre-built lidar maps. To bridge the gap of sensing modalities, deep neural networks are constructed to create shared embedding space for radar scans and lidar maps. Herein learned feature embeddings are supportive for similarity measurement, thus improving map retrieval and data matching respectively. In RobotCar and MulRan datasets, we demonstrate the effectiveness of the proposed framework with the comparison to Scan Context and RaLL. In addition, the proposed pose tracking pipeline i…
4D-CAAL: 4D Radar-Camera Calibration and Auto-Labeling for Autonomous Driving
School of Information Engineering, Yancheng Institute of Technology（信息工程学院，盐城职业技术学院）；School of Navigation, Wuhan University of Technology（导航学院，武汉理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；School of Information Engineering, Yancheng Institute Technology（信息工程学院，盐城职业技术学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（先进技术学院，西安交通大学利物浦大学）；School of Information Science and Technology, Nantong University（信息科学与技术学院，南通大学）；State Key Laboratory of Maritime Technology and Safety（船舶技术与安全国家重点实验室）
