Sensor fusion is essential for autonomous driving and autonomous robots, and radar-camera fusion systems have gained popularity due to their complementary sensing capabilities. However, accurate calibration between these two sensors is crucial to ensure effective fusion and improve overall system performance. Calibration involves intrinsic and extrinsic calibration, with the latter being particularly important for achieving accurate sensor fusion. Unfortunately, many target-based calibration methods require complex operating procedures and well-designed experimental conditions, posing challenges for researchers attempting to reproduce the results. To address this issue, we introduce a novel approach that leverages deep learning to extract a common feature from raw radar data (i.e., Range-Doppler-Angle data) and camera images. Instead of explicitly representing these common features, our method implicitly utilizes these common features to match identical objects from both data sources. Specifically, the extracted common feature serves as an example to demonstrate an online targetless calibration method between the radar and camera systems. The estimation of the extrinsic transformation matrix is achieved through this feature-based approach. To enhance the accuracy and robustness of the calibration, we apply the RANSAC and Levenberg-Marquardt (LM) nonlinear optimization algorithm for deriving the matrix. Our experiments in the real world demonstrate the effectiveness and accuracy of our proposed method.
Multi-LiCa: A Motion and Targetless Multi LiDAR-to-LiDAR Calibration Framework
Institute of Automotive Technology（汽车技术研究所）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与智能机械研究所）；Technical University of Munich（慕尼黑技术大学）
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）；Vector Institute（Vector研究所）
Reliable operation in inclement weather is essential to the deployment of safe autonomous vehicles (AVs). Robustness and reliability can be achieved by fusing data from the standard AV sensor suite (i.e., lidars, cameras) with weather robust sensors, such as millimetre-wavelength radar. Critically, accurate sensor data fusion requires knowledge of the rigid-body transform between sensor pairs, which can be determined through the process of extrinsic calibration. A number of extrinsic calibration algorithms have been designed for 2D (planar) radar sensors - however, recently-developed, low-cost…
A 4D Radar Camera Extrinsic Calibration Tool Based on 3D Uncertainty Perspective N Points
School of Automation and Intelligent Sensing, Institute of Medical Robotics, Shanghai Jiao Tong University（自动化与智能感知学院、医疗机器人研究院、上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240（系统控制与信息处理重点实验室、中华人民共和国教育部、上海200240）；Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（瑞金医院、上海交通大学医学院）；Renji hospital, Shanghai Jiao Tong University School of Medicine（仁济医院、上海交通大学医学院）
The University of Hong Kong（香港大学）；Southern University of Science and Technology（南方科技大学）
In this letter, we present a novel method for automatic extrinsic calibration of high-resolution LiDARs and RGB cameras in targetless environments. Our approach does not require checkerboards but can achieve pixel-level accuracy by aligning natural edge features in the two sensors. On the theory level, we analyze the constraints imposed by edge features and the sensitivity of calibration accuracy with respect to edge distribution in the scene. On the implementation level, we carefully investigate the physical measuring principles of LiDARs and propose an efficient and accurate LiDAR edge extra…
MFCalib: Single-shot and Automatic Extrinsic Calibration for LiDAR and Camera in Targetless Environments Based on Multi-Feature Edge
College of Mechatronics and Control Engineering, Shenzhen University（深圳大学机电与控制工程学院）；Peng Cheng Laboratory（鹏城实验室）；Manifold Tech Limited（万孚科技有限公司）；The University of Hong Kong（香港大学）
Determining the extrinsic parameter between multiple LiDARs and cameras is essential for autonomous robots, especially for solid-state LiDARs, where each LiDAR unit has a very small Field-of-View (FoV), and multiple units are often used collectively. The majority of extrinsic calibration methods are proposed for 360$^\circ$ mechanical spinning LiDARs where the FoV overlap with other LiDAR or camera sensors is assumed. Few research works have been focused on the calibration of small FoV LiDARs and cameras nor on the improvement of the calibration speed. In this work, we consider the problem of …
Joint Optimization-based Targetless Extrinsic Calibration for Multiple LiDARs and GNSS-Aided INS of Ground Vehicles
Institute of Systems Engineering and Collaborative Laboratory for Intelligent Science and Systems, Macau University of Science and Technology（系统工程研究院和智能科学与系统联合实验室，澳门科学技术大学）；Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）
Achieving safe and reliable autonomous driving relies greatly on the ability to achieve an accurate and robust perception system; however, this cannot be fully realized without precisely calibrated sensors. Environmental and operational conditions as well as improper maintenance can produce calibration errors inhibiting sensor fusion and, consequently, degrading the perception performance. Traditionally, sensor calibration is performed in a controlled environment with one or more known targets. Such a procedure can only be carried out in between drives and requires manual operation; a tedious …
4D-CAAL: 4D Radar-Camera Calibration and Auto-Labeling for Autonomous Driving
School of Information Engineering, Yancheng Institute of Technology（信息工程学院，盐城职业技术学院）；School of Navigation, Wuhan University of Technology（导航学院，武汉理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；School of Information Engineering, Yancheng Institute Technology（信息工程学院，盐城职业技术学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（先进技术学院，西安交通大学利物浦大学）；School of Information Science and Technology, Nantong University（信息科学与技术学院，南通大学）；State Key Laboratory of Maritime Technology and Safety（船舶技术与安全国家重点实验室）
4D Millimeter-Wave Radar in Autonomous Driving: A Survey
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）；Xingjian College, Tsinghua University（清华大学行健书院）；Changan Automobile Co., Ltd.（长安汽车股份有限公司）
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学工业控制研究所）
Sensor calibration is the fundamental block for a multi-sensor fusion system. This paper presents an accurate and repeatable LiDAR-IMU calibration method (termed LI-Calib), to calibrate the 6-DOF extrinsic transformation between the 3D LiDAR and the Inertial Measurement Unit (IMU). % Regarding the high data capture rate for LiDAR and IMU sensors, LI-Calib adopts a continuous-time trajectory formulation based on B-Spline, which is more suitable for fusing high-rate or asynchronous measurements than discrete-time based approaches. % Additionally, LI-Calib decomposes the space into cells and iden…
This paper presents a framework for the targetless extrinsic calibration of stereo cameras and Light Detection and Ranging (LiDAR) sensors with a non-overlapping Field of View (FOV). In order to solve the extrinsic calibrations problem under such challenging configuration, the proposed solution exploits road markings as static and robust features among the various dynamic objects that are present in urban environment. First, this study utilizes road markings that are commonly captured by the two sensor modalities to select informative images for estimating the extrinsic parameters. In order to…
Radar-Inertial Odometry with Online Spatio-Temporal Calibration via Continuous-Time IMU Modeling
University of Zagreb Faculty of Electrical Engineering and Computing, Laboratory for Autonomous Systems and Mobile Robotics (LAMOR)（泽格布大学电子工程与计算学院，自主系统与移动机器人实验室）；Calirad d.o.o.（Calirad公司）
RAVES-Calib: Robust, Accurate and Versatile Extrinsic Self Calibration Using Optimal Geometric Features
School of Systems Science and Engineering, Sun Yat-Sen University（系统科学与工程学院，中山大学）；Defense Innovation Institute, Chinese Academy of Military Science（国防创新研究院，中国人民解放军军事科学院）；Intelligent Game and Decision Laboratory（智能游戏与决策实验室）；College of Computer Science and Technology, Harbin Engineering University（计算机科学与技术学院，哈尔滨工程大学）
Institute of Industrial Science, The University of Tokyo（东京大学产业技术研究所）；Microsoft（微软）
In this paper, we propose a method of targetless and automatic Camera-LiDAR calibration. Our approach is an extension of hand-eye calibration framework to 2D-3D calibration. By using the sensor fusion odometry method, the scaled camera motions are calculated with high accuracy. In addition to this, we clarify the suitable motion for this calibration method. The proposed method only requires the three-dimensional point cloud and the camera image and does not need other information such as reflectance of LiDAR and to give initial extrinsic parameter. In the experiments, we demonstrate our method…
Accurate depth-sensing plays a crucial role in securing a high success rate of robotic harvesting in natural orchard environments. Solid-state LiDAR (SSL), a recently introduced LiDAR technique, can perceive high-resolution geometric information of the scenes, which can be potential utilised to receive accurate depth information. Meanwhile, the fusion of the sensory information from LiDAR and camera can significantly enhance the sensing ability of the harvesting robots. This work introduces a LiDAR-camera fusion-based visual sensing and perception strategy to perform accurate fruit localisatio…
Southern University of Science and Technology（南方科技大学）；Zhejiang University（浙江大学）
This paper presents a novel 3D mapping robot with an omnidirectional field-of-view (FoV) sensor suite composed of a non-repetitive LiDAR and an omnidirectional camera. Thanks to the non-repetitive scanning nature of the LiDAR, an automatic targetless co-calibration method is proposed to simultaneously calibrate the intrinsic parameters for the omnidirectional camera and the extrinsic parameters for the camera and LiDAR, which is crucial for the required step in bringing color and texture information to the point clouds in surveying and mapping tasks. Comparisons and analyses are made to target…
Mobile robotic applications need precise information about the geometric position of the individual sensors on the platform. This information is given by the extrinsic calibration parameters which define how the sensor is rotated and translated with respect to a fixed reference coordinate system. Erroneous calibration parameters have a negative impact on typical robotic estimation tasks, e.g. SLAM. In this work we propose a new method for a continuous estimation of the calibration parameters during operation of the robot. The parameter estimation is based on the matching of point clouds which …
Doppler-SLAM: Doppler-Aided Radar-Inertial and LiDAR-Inertial Simultaneous Localization and Mapping
Julius-Maximilians-Universität Würzburg（乌尔姆-马克斯-普朗克大学）；Zentrum für Telematik e.V.（电信研究中心）；Nuremberg Institute of Technology Georg Simon Ohm（纽伦堡技术学院 Georg Simon Ohm）；CARIAD SE（CARIAD SE公司）；University of Bonn（波恩大学）；Center for Robotics（机器人中心）；Lamarr Institute for Machine Learning and Artificial Intelligence（Lamarr机器学习与人工智能研究所）
