This paper presents a novel semantic-based online extrinsic calibration approach, SOIC (so, I see), for Light Detection and Ranging (LiDAR) and camera sensors. Previous online calibration methods usually need prior knowledge of rough initial values for optimization. The proposed approach removes this limitation by converting the initialization problem to a Perspective-n-Point (PnP) problem with the introduction of semantic centroids (SCs). The closed-form solution of this PnP problem has been well researched and can be found with existing PnP methods. Since the semantic centroid of the point cloud usually does not accurately match with that of the corresponding image, the accuracy of parameters are not improved even after a nonlinear refinement process. Thus, a cost function based on the constraint of the correspondence between semantic elements from both point cloud and image data is formulated. Subsequently, optimal extrinsic parameters are estimated by minimizing the cost function. We evaluate the proposed method either with GT or predicted semantics on KITTI dataset. Experimental results and comparisons with the baseline method verify the feasibility of the initialization strategy and the accuracy of the calibration approach. In addition, we release the source code at https://github.com/--/SOIC.
Transforming Omnidirectional RGB-LiDAR data into 3D Gaussian Splatting
Department of Computer Science, State University of New York（计算机科学系，纽约州立大学）；Department of Mechanical Engineering, State University of New York（机械工程系，纽约州立大学）
The University of Adelaide（阿德莱德大学）；ETH Zurich（苏黎世联邦理工学院）
Point cloud registration is a fundamental problem in 3D scanning. In this paper, we address the frequent special case of registering terrestrial LiDAR scans (or, more generally, levelled point clouds). Many current solutions still rely on the Iterative Closest Point (ICP) method or other heuristic procedures, which require good initializations to succeed and/or provide no guarantees of success. On the other hand, exact or optimal registration algorithms can compute the best possible solution without requiring initializations; however, they are currently too slow to be practical in realistic ap…
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）；CWB Institute of Autonomous Driving（CWB自动驾驶研究院）
The combination of LiDARs and cameras enables a mobile robot to perceive environments with multi-modal data, becoming a key factor in achieving robust perception. Traditional frame cameras are sensitive to changing illumination conditions, motivating us to introduce novel event cameras to make LiDAR-camera fusion more complete and robust. However, to jointly exploit these sensors, the challenging extrinsic calibration problem should be addressed. This paper proposes an automatic checkerboard-based approach to calibrate extrinsics between a LiDAR and a frame/event camera, where four contributio…
LiDAR-VGGT: Cross-Modal Coarse-to-Fine Fusion for Globally Consistent and Metric-Scale Dense Mapping
State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University（工业控制技术国家重点实验室，系统与控制研究院，浙江大学）；Differential Robot Technology Co., Ltd.（差分机器人技术有限公司）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）
Targetless LiDAR-Camera Calibration with Neural Gaussian Splatting
Department of Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能交叉学科项目系，首尔国立大学）；Department of Artificial Intelligence, Yonsei University（人工智能系，延世大学）
University of Rochester（罗切斯特大学）；NYU Tandon School of Engineering（纽约大学坦登工程学院）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））
We propose DeepMapping, a novel registration framework using deep neural networks (DNNs) as auxiliary functions to align multiple point clouds from scratch to a globally consistent frame. We use DNNs to model the highly non-convex mapping process that traditionally involves hand-crafted data association, sensor pose initialization, and global refinement. Our key novelty is that "training" these DNNs with properly defined unsupervised losses is equivalent to solving the underlying registration problem, but less sensitive to good initialization than ICP. Our framework contains two DNNs: a locali…
This work studies semantic segmentation using 3D LiDAR data. Popular deep learning methods applied for this task require a large number of manual annotations to train the parameters. We propose a new method that makes full use of the advantages of traditional methods and deep learning methods via incorporating human domain knowledge into the neural network model to reduce the demand for large numbers of manual annotations and improve the training efficiency. We first pretrain a model with autogenerated samples from a rule-based classifier so that human knowledge can be propagated into the netw…
The Hong Kong University of Science and Technology（香港科技大学）
Multiple LiDARs have progressively emerged on autonomous vehicles for rendering a wide field of view and dense measurements. However, the lack of precise calibration negatively affects their potential applications in localization and perception systems. In this paper, we propose a novel system that enables automatic multi-LiDAR calibration without any calibration target, prior environmental information, and initial values of the extrinsic parameters. Our approach starts with a hand-eye calibration for automatic initialization by aligning the estimated motions of each sensor. The resulting para…
Hong Kong University of Science and Technology（香港科技大学）
Combining multiple LiDARs enables a robot to maximize its perceptual awareness of environments and obtain sufficient measurements, which is promising for simultaneous localization and mapping (SLAM). This paper proposes a system to achieve robust and simultaneous extrinsic calibration, odometry, and mapping for multiple LiDARs. Our approach starts with measurement preprocessing to extract edge and planar features from raw measurements. After a motion and extrinsic initialization procedure, a sliding window-based multi-LiDAR odometry runs onboard to estimate poses with online calibration refine…
Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations
School of Automation, Beijing Institute of Technology（北京理工大学自动化学院）；School of Integrated Circuits and Electronics, Beijing Institute of Technology（北京理工大学集成电路与电子学院）
ForestMamba: Sparse Mamba with Geometry-guided Queries for 3D Forest Point Cloud Segmentation
Nagoya University（名古屋大学）；RIKEN Seika（日本理化学研究所Seika研究中心）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Twente（埃因霍温理工大学）；Ritsumeikan University（立命馆大学）
German Research Center for Artificial Intelligence（德国人工智能研究中心）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
This article introduces a new physics-based method for rigid point set alignment called Fast Gravitational Approach (FGA). In FGA, the source and target point sets are interpreted as rigid particle swarms with masses interacting in a globally multiply-linked manner while moving in a simulated gravitational force field. The optimal alignment is obtained by explicit modeling of forces acting on the particles as well as their velocities and displacements with second-order ordinary differential equations of motion. Additional alignment cues (point-based or geometric features, and other boundary co…
Uber Advanced Technologies Group（优步先进技术集团）；University of Toronto（多伦多大学）
In the past few years we have seen great advances in object perception (particularly in 4D space-time dimensions) thanks to deep learning methods. However, they typically rely on large amounts of high-quality labels to achieve good performance, which often require time-consuming and expensive work by human annotators. To address this we propose an automatic annotation pipeline that generates accurate object trajectories in 3D space (i.e., 4D labels) from LiDAR point clouds. The key idea is to decompose the 4D object label into two parts: the object size in 3D that's fixed through time for rigi…
The Hong Kong University of Science and Technology（香港科技大学）；Unity-Drive technology Inc（Unity-Drive科技公司）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Multiple lidars are prevalently used on mobile vehicles for rendering a broad view to enhance the performance of localization and perception systems. However, precise calibration of multiple lidars is challenging since the feature correspondences in scan points cannot always provide enough constraints. To address this problem, the existing methods require fixed calibration targets in scenes or rely exclusively on additional sensors. In this paper, we present a novel method that enables automatic lidar calibration without these restrictions. Three linearly independent planar surfaces appearing …
Valeo Schalter und Sensoren GmbH（法雷奥开关与传感器有限公司）；Spleenlab GmbH（Spleenlab有限公司）；Ilmenau University of Technology（伊尔梅瑙工业大学）；Software Engineering for Safety-Critical Systems（安全关键系统软件工程）
Robust point cloud registration in real-time is an important prerequisite for many mapping and localization algorithms. Traditional methods like ICP tend to fail without good initialization, insufficient overlap or in the presence of dynamic objects. Modern deep learning based registration approaches present much better results, but suffer from a heavy run-time. We overcome these drawbacks by introducing StickyPillars, a fast, accurate and extremely robust deep middle-end 3D feature matching method on point clouds. It uses graph neural networks and performs context aggregation on sparse 3D key…
Segment Any Point Cloud Sequences by Distilling Vision Foundation Models
Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）
HGS-Mapping: Online Dense Mapping Using Hybrid Gaussian Representation in Urban Scenes
Fudan University（复旦大学）；School of Future Technology, Harbin Institute of Technology（哈尔滨工业大学未来技术学院）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）
Hong Kong University of Science and Technology（香港科技大学）；Huawei（华为）；City University of Hong Kong（香港城市大学）
LiDAR and camera are two important sensors for 3D object detection in autonomous driving. Despite the increasing popularity of sensor fusion in this field, the robustness against inferior image conditions, e.g., bad illumination and sensor misalignment, is under-explored. Existing fusion methods are easily affected by such conditions, mainly due to a hard association of LiDAR points and image pixels, established by calibration matrices. We propose TransFusion, a robust solution to LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions. Specifically, our Trans…
AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；DarkMatter AI Research（DarkMatter AI研究）；University of Oxford（牛津大学）；HKU（香港大学）；Cainiao Group（菜鸟集团）；NCUT（北方工业大学）；Sun Yat-sen University（中山大学）；Westlake University（西湖大学）
