V2I-Calib: A Novel Calibration Approach for Collaborative Vehicle and Infrastructure LiDAR Systems
China University of Mining and Technology -Beijing（中国矿业大学（北京））；Tsinghua University（清华大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
整合车辆与道路基础设施的协同激光雷达系统（即V2I标定）展现出巨大潜力，但其部署仍面临诸多挑战。确保这类系统数据准确性与一致性的一个关键环节，是异构车辆端与基础设施端激光雷达单元的标定。这就需要开发兼具实时性与鲁棒性的标定方法，尤其是无需依赖初始定位值、能在城市峡谷场景下保持稳定性能的方法。为此，本文提出一种新型V2I标定方法，该方法利用感知目标间的空间关联信息。该方法的核心是创新的整体交并比（Overall Intersection over Union，oIoU）指标，用于量化车辆与基础设施系统识别目标间的关联度，从而实现对标定结果的实时监测。我们的方法通过构建亲和矩阵，在车辆与基础设施激光雷达系统的感知结果中识别共同目标，再以这些共同目标为基础计算并优化外参。在DAIR-V2X数据集上开展的对比实验与消融实验验证了所提方法的优越性。更多见解与资源可访问我们的项目仓库：https://github.com/MassimoQu/v2i-calib。
英文摘要：
Cooperative LiDAR systems integrating vehicles and road infrastructure, termed V2I calibration, exhibit substantial potential, yet their deployment encounters numerous challenges. A pivotal aspect of ensuring data accuracy and consistency across such systems involves the calibration of LiDAR units across heterogeneous vehicular and infrastructural endpoints. This necessitates the development of calibration methods that are both real-time and robust, particularly those that can ensure robust performance in urban canyon scenarios without relying on initial positioning values. Accordingly, this paper introduces a novel approach to V2I calibration, leveraging spatial association information among perceived objects. Central to this method is the innovative Overall Intersection over Union (oIoU) metric, which quantifies the correlation between targets identified by vehicle and infrastructure systems, thereby facilitating the real-time monitoring of calibration results. Our approach involves identifying common targets within the perception results of vehicle and infrastructure LiDAR systems through the construction of an affinity matrix. These common targets then form the basis for the calculation and optimization of extrinsic parameters. Comparative and ablation studies conducted using the DAIR-V2X dataset substantiate the superiority of our approach. For further insights and resources, our project repository is accessible at https://github.com/MassimoQu/v2i-calib.
Collaborative Learning for Semi-Supervised LiDAR Semantic Segmentation
Bosch Research, Robert Bosch GmbH, Stuttgart, Germany（博世研究, 博世有限公司, 斯图加特, 德国）；Institute for Neuro- and Bioinformatics, University of Lübeck, Lübeck, Germany（神经与生物信息学研究所, 奥尔德马克大学, 奥尔德马克, 德国）
This work proposes a novel motion guided method for target-less self-calibration of a LiDAR and camera and use the re-projection of LiDAR points onto the image reference frame for real-time depth upsampling. The calibration parameters are estimated by optimizing an objective function that penalizes distances between 2D and re-projected 3D motion vectors obtained from time-synchronized image and point cloud sequences. For upsampling, a simple, yet effective and time efficient formulation that minimizes depth gradients subject to an equality constraint involving the LiDAR measurements is propose…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；FZI Research Center for Information Technology（FZI信息技术研究中心）
In this paper, we present TEScalib, a novel extrinsic self-calibration approach of LiDAR and stereo camera using the geometric and photometric information of surrounding environments without any calibration targets for automated driving vehicles. Since LiDAR and stereo camera are widely used for sensor data fusion on automated driving vehicles, their extrinsic calibration is highly important. However, most of the LiDAR and stereo camera calibration approaches are mainly target-based and therefore time consuming. Even the newly developed targetless approaches in last years are either inaccurate…
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Sensor-based environmental perception is a crucial part of the autonomous driving system. In order to get an excellent perception of the surrounding environment, an intelligent system would configure multiple LiDARs (3D Light Detection and Ranging) to cover the distant and near space of the car. The precision of perception relies on the quality of sensor calibration. This research aims at developing an accurate, automatic, and robust calibration strategy for multiple LiDAR systems in the general road scene. We thus propose CROON (automatiC multi-LiDAR CalibratiOn and Refinement method in rOad …
In this paper, we propose a novel online self-calibration approach for Light Detection and Ranging (LiDAR) and camera sensors. Compared to the previous CNN-based methods that concatenate the feature maps of the RGB image and decalibrated depth image, we exploit the cost volume inspired by the PWC-Net for feature matching. Besides the smooth L1-Loss of the predicted extrinsic calibration parameters, an additional point cloud loss is applied. Instead of regress the extrinsic parameters between LiDAR and camera directly, we predict the decalibrated deviation from initial calibration to the ground…
Texas A&M University（德克萨斯A&M大学）；DEVCOM Army Research Laboratory (ARL)（DEVCOM陆军研究实验室）
We propose an algorithm for automatic, targetless, extrinsic calibration of a LiDAR and camera system using semantic information. We achieve this goal by maximizing mutual information (MI) of semantic information between sensors, leveraging a neural network to estimate semantic mutual information, and matrix exponential for calibration computation. Using kernel-based sampling to sample data from camera measurement based on LiDAR projected points, we formulate the problem as a novel differentiable objective function which supports the use of gradient-based optimization methods. We also introduc…
Mercedes-Benz AG（梅赛德斯-奔驰股份公司）；University of Freiburg（弗莱堡大学）
The calibration of deep learning-based perception models plays a crucial role in their reliability. Our work focuses on a class-wise evaluation of several model's confidence performance for LiDAR-based semantic segmentation with the aim of providing insights into the calibration of underrepresented classes. Those classes often include VRUs and are thus of particular interest for safety reasons. With the help of a metric based on sparsification curves we compare the calibration abilities of three semantic segmentation models with different architectural concepts, each in a in deterministic and …
The Hong Kong University of Science and Technology（香港科技大学）；Unity-Drive technology Inc（Unity-Drive科技公司）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Multiple lidars are prevalently used on mobile vehicles for rendering a broad view to enhance the performance of localization and perception systems. However, precise calibration of multiple lidars is challenging since the feature correspondences in scan points cannot always provide enough constraints. To address this problem, the existing methods require fixed calibration targets in scenes or rely exclusively on additional sensors. In this paper, we present a novel method that enables automatic lidar calibration without these restrictions. Three linearly independent planar surfaces appearing …
CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning
Hungarian Research Network Institute for Computer Science and Control (HUN-REN SZTAKI)（匈牙利研究网络计算机科学与控制研究所）；Pázmány Péter Catholic University（帕兹曼·佩斯特天主教大学）；Intelligent Vehicles Group, TU Delft（智能车辆组，代尔夫特理工大学）；Perciv AI
Targetless LiDAR-Camera Calibration with Neural Gaussian Splatting
Department of Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能交叉学科项目系，首尔国立大学）；Department of Artificial Intelligence, Yonsei University（人工智能系，延世大学）
Beihang University（北京航空航天大学）；Beijing Advanced Innovation Center for Big Data and Brain Computing（大数据与脑计算北京先进创新中心）；Hangzhou Innovation Institution, Beihang University（北京航空航天大学杭州创新研究院）
Recently, the rapid development of Solid-State LiDAR (SSL) enables low-cost and efficient obtainment of 3D point clouds from the environment, which has inspired a large quantity of studies and applications. However, the non-uniformity of its scanning pattern, and the inconsistency of the ranging error distribution bring challenges to its calibration task. In this paper, we proposed a fully automatic calibration method for the non-repetitive scanning SSL and camera systems. First, a temporal-spatial-based geometric feature refinement method is presented, to extract effective features from SSL p…
Global-Local Collaborative Inference with LLM for Lidar-Based Open-Vocabulary Detection
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Meituan（美团）；University of Science and Technology Beijing（北京科技大学）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）
Texas A&M University（得克萨斯农工大学）；DEVCOM Army Research Laboratory (ARL)（陆军研究实验室）
This paper proposes SemCal: an automatic, targetless, extrinsic calibration algorithm for a LiDAR and camera system using semantic information. We leverage a neural information estimator to estimate the mutual information (MI) of semantic information extracted from each sensor measurement, facilitating semantic-level data association. By using a matrix exponential formulation of the $se(3)$ transformation and a kernel-based sampling method to sample from camera measurement based on LiDAR projected points, we can formulate the LiDAR-Camera calibration problem as a novel differentiable objective…
CoLC: Communication-Efficient Collaborative Perception with LiDAR Completion
Key Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education（交通运输大数据与人工智能重点实验室，教育部）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）；Xiamen University（厦门大学）
In view of contemporary panoramic camera-laser scanner system, the traditional calibration method is not suitable for panoramic cameras whose imaging model is extremely nonlinear. The method based on statistical optimization has the disadvantage that the requirement of the number of laser scanner's channels is relatively high. Calibration equipments with extreme accuracy for panoramic camera-laser scanner system are costly. Facing all these in the calibration of panoramic camera-laser scanner system, a method based on supervised learning is proposed. Firstly, corresponding feature points of pa…
LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance
University of Science and Technology of China（中国科学技术大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）
As neuromorphic technology is maturing, its application to robotics and autonomous vehicle systems has become an area of active research. In particular, event cameras have emerged as a compelling alternative to frame-based cameras in low-power and latency-demanding applications. To enable event cameras to operate alongside staple sensors like lidar in perception tasks, we propose a direct, temporally-decoupled extrinsic calibration method between event cameras and lidars. The high dynamic range, high temporal resolution, and low-latency operation of event cameras are exploited to directly regi…
The Hong Kong University of Science and Technology（香港科技大学）
Multiple LiDARs have progressively emerged on autonomous vehicles for rendering a wide field of view and dense measurements. However, the lack of precise calibration negatively affects their potential applications in localization and perception systems. In this paper, we propose a novel system that enables automatic multi-LiDAR calibration without any calibration target, prior environmental information, and initial values of the extrinsic parameters. Our approach starts with a hand-eye calibration for automatic initialization by aligning the estimated motions of each sensor. The resulting para…
This paper proposes a simple self-calibration method for the internal time synchronization of MEMS(Micro-electromechanical systems) LiDAR during research and development. Firstly, we introduced the problem of internal time misalignment in MEMS lidar. Then, a robust Minimum Vertical Gradient(MVG) prior is proposed to calibrate the time difference between the laser and MEMS mirror, which can be calculated automatically without any artificial participation or specially designed cooperation target. Finally, actual experiments on MEMS LiDARs are implemented to demonstrate the effectiveness of the p…
Institute of Industrial Science, The University of Tokyo（东京大学产业技术研究所）；Microsoft（微软）
In this paper, we propose a method of targetless and automatic Camera-LiDAR calibration. Our approach is an extension of hand-eye calibration framework to 2D-3D calibration. By using the sensor fusion odometry method, the scaled camera motions are calculated with high accuracy. In addition to this, we clarify the suitable motion for this calibration method. The proposed method only requires the three-dimensional point cloud and the camera image and does not need other information such as reflectance of LiDAR and to give initial extrinsic parameter. In the experiments, we demonstrate our method…
Hong Kong University of Science and Technology（香港科技大学）
LiDAR and cameras are two complementary sensors for 3D perception in autonomous driving. LiDAR point clouds have accurate spatial and geometry information, while RGB images provide textural and color data for context reasoning. To exploit LiDAR and cameras jointly, existing fusion methods tend to align each 3D point to only one projected image pixel based on calibration, namely one-to-one mapping. However, the performance of these approaches highly relies on the calibration quality, which is sensitive to the temporal and spatial synchronization of sensors. Therefore, we propose a Dynamic Cross…
Sensor calibration, which can be intrinsic or extrinsic, is an essential step to achieve the measurement accuracy required for modern perception and navigation systems deployed on autonomous robots. To date, intrinsic calibration models for spinning LiDARs have been based on hypothesized based on their physical mechanisms, resulting in anywhere from three to ten parameters to be estimated from data, while no phenomenological models have yet been proposed for solid-state LiDARs. Instead of going down that road, we propose to abstract away from the physics of a LiDAR type (spinning vs solid-stat…
Vellore Institute of Technology, Chennai（金奈韦洛尔理工大学）；Birla Institute of Technology and Science, Hyderabad（海得拉巴比尔拉理工学院）；Veermata Jijabai Technological Institute, Mumbai（孟买韦尔马塔·吉贾拜技术学院）；International Institute of Information and Technology, Hyderabad（海得拉巴国际信息技术学院）
With the advent of autonomous vehicles, LiDAR and cameras have become an indispensable combination of sensors. They both provide rich and complementary data which can be used by various algorithms and machine learning to sense and make vital inferences about the surroundings. We propose a novel pipeline and experimental setup to find accurate rigid-body transformation for extrinsically calibrating a LiDAR and a camera. The pipeling uses 3D-3D point correspondences in LiDAR and camera frame and gives a closed form solution. We further show the accuracy of the estimate by fusing point clouds fro…
Arizona State University（亚利桑那州立大学）；U.S. Army DEVCOM Research Laboratory（美国陆军作战能力发展司令部研究实验室）；U.S. Army DEVCOM C5ISR Center（美国陆军作战能力发展司令部指挥、控制、通信、计算机、网络、情报、监视与侦察中心）
Motivated by the increasing application of low-resolution LiDAR recently, we target the problem of low-resolution LiDAR-camera calibration in this work. The main challenges are two-fold: sparsity and noise in point clouds. To address the problem, we propose to apply depth interpolation to increase the point density and supervised contrastive learning to learn noise-resistant features. The experiments on RELLIS-3D demonstrate that our approach achieves an average mean absolute rotation/translation errors of 0.15cm/0.33\textdegree on 32-channel LiDAR point cloud data, which significantly outperf…
Light Detection and Ranging (LIDAR) sensors play an important role in the perception stack of autonomous robots, supplying mapping and localization pipelines with depth measurements of the environment. While their accuracy outperforms other types of depth sensors, such as stereo or time-of-flight cameras, the accurate modeling of LIDAR sensors requires laborious manual calibration that typically does not take into account the interaction of laser light with different surface types, incidence angles and other phenomena that significantly influence measurements. In this work, we introduce a phys…
With information from multiple input modalities, sensor fusion-based algorithms usually out-perform their single-modality counterparts in robotics. Camera and LIDAR, with complementary semantic and depth information, are the typical choices for detection tasks in complicated driving environments. For most camera-LIDAR fusion algorithms, however, the calibration of the sensor suite will greatly impact the performance. More specifically, the detection algorithm usually requires an accurate geometric relationship among multiple sensors as the input, and it is often assumed that the contents from …
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）；CWB Institute of Autonomous Driving（CWB自动驾驶研究院）
The combination of LiDARs and cameras enables a mobile robot to perceive environments with multi-modal data, becoming a key factor in achieving robust perception. Traditional frame cameras are sensitive to changing illumination conditions, motivating us to introduce novel event cameras to make LiDAR-camera fusion more complete and robust. However, to jointly exploit these sensors, the challenging extrinsic calibration problem should be addressed. This paper proposes an automatic checkerboard-based approach to calibrate extrinsics between a LiDAR and a frame/event camera, where four contributio…
Iterative Camera-LiDAR Extrinsic Optimization via Surrogate Diffusion
School of Automation, Beijing Institute of Technology（自动化学院，北京理工大学）；Centre for Robotics Research, Department of Engineering, King’s College London（机器人研究中心，工程系，伦敦大学国王学院）；School of Integrated Circuits and Electronics, Beijing Institute of Technology（集成电路与电子学院，北京理工大学）
For autonomous vehicles, an accurate calibration for LiDAR and camera is a prerequisite for multi-sensor perception systems. However, existing calibration techniques require either a complicated setting with various calibration targets, or an initial calibration provided beforehand, which greatly impedes their applicability in large-scale autonomous vehicle deployment. To tackle these issues, we propose a novel method to calibrate the extrinsic parameter for LiDAR and camera in road scenes. Our method introduces line features from static straight-line-shaped objects such as road lanes and pole…
The Institute of Industrial Science, The University of Tokyo（东京大学生产技术研究所）；The National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）
Australian Centre for Field Robotics（澳大利亚野外机器人学中心）；University of Sydney（悉尼大学）
This paper proposes an automated method to obtain the extrinsic calibration parameters between a camera and a 3D lidar with as low as 16 beams. We use a checkerboard as a reference to obtain features of interest in both sensor frames. The calibration board centre point and normal vector are automatically extracted from the lidar point cloud by exploiting the geometry of the board. The corresponding features in the camera image are obtained from the camera's extrinsic matrix. We explain the reasons behind selecting these features, and why they are more robust compared to other possibilities. To…
Hong Kong University of Science and Technology（香港科技大学）
Combining multiple LiDARs enables a robot to maximize its perceptual awareness of environments and obtain sufficient measurements, which is promising for simultaneous localization and mapping (SLAM). This paper proposes a system to achieve robust and simultaneous extrinsic calibration, odometry, and mapping for multiple LiDARs. Our approach starts with measurement preprocessing to extract edge and planar features from raw measurements. After a motion and extrinsic initialization procedure, a sliding window-based multi-LiDAR odometry runs onboard to estimate poses with online calibration refine…
UniCalib: Targetless LiDAR-Camera Calibration via Probabilistic Flow on Unified Depth Representations
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
