Building accurate maps is a key building block to enable reliable localization, planning, and navigation of autonomous vehicles. We propose a novel approach for building accurate maps of dynamic environments utilizing a sequence of LiDAR scans. To this end, we propose encoding the 4D scene into a novel spatio-temporal implicit neural map representation by fitting a time-dependent truncated signed distance function to each point. Using our representation, we extract the static map by filtering the dynamic parts. Our neural representation is based on sparse feature grids, a globally shared decoder, and time-dependent basis functions, which we jointly optimize in an unsupervised fashion. To learn this representation from a sequence of LiDAR scans, we design a simple yet efficient loss function to supervise the map optimization in a piecewise way. We evaluate our approach on various scenes containing moving objects in terms of the reconstruction quality of static maps and the segmentation of dynamic point clouds. The experimental results demonstrate that our method is capable of removing the dynamic part of the input point clouds while reconstructing accurate and complete 3D maps, outperforming several state-of-the-art methods. Codes are available at: https://github.com/PRBonn/4dNDF
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；The University of Hong Kong（香港大学）；Hong Kong University of Science and Technology（香港科技大学）；DJI Co.（大疆创新科技有限公司）
Modeling scene geometry using implicit neural representation has revealed its advantages in accuracy, flexibility, and low memory usage. Previous approaches have demonstrated impressive results using color or depth images but still have difficulty handling poor light conditions and large-scale scenes. Methods taking global point cloud as input require accurate registration and ground truth coordinate labels, which limits their application scenarios. In this paper, we propose a new method that uses sparse LiDAR point clouds and rough odometry to reconstruct fine-grained implicit occupancy field…
HD (High Definition) map based on 3D lidar plays a vital role in autonomous vehicle localization, planning, decision-making, perception, etc. Many 3D lidar mapping technologies related to SLAM (Simultaneous Localization and Mapping) are used in HD map construction to ensure its high accuracy. To evaluate the accuracy of 3D lidar mapping, the most common methods use ground truth of poses to calculate the error between estimated poses and ground truth, however it's usually so difficult to get the ground truth of poses in the actual lidar mapping for autonomous vehicle. In this paper, we proposed…
This paper presents a loop closure method to correct the long-term drift in LiDAR odometry and mapping (LOAM). Our proposed method computes the 2D histogram of keyframes, a local map patch, and uses the normalized cross-correlation of the 2D histograms as the similarity metric between the current keyframe and those in the map. We show that this method is fast, invariant to rotation, and produces reliable and accurate loop detection. The proposed method is implemented with careful engineering and integrated into the LOAM algorithm, forming a complete and practical system ready to use. To benefi…
Shanghai Jiao Tong University（上海交通大学）；College of Intelligence Science and Technology, NUDT（国防科技大学智能科学与技术学院）
Simultaneously odometry and mapping using LiDAR data is an important task for mobile systems to achieve full autonomy in large-scale environments. However, most existing LiDAR-based methods prioritize tracking quality over reconstruction quality. Although the recently developed neural radiance fields (NeRF) have shown promising advances in implicit reconstruction for indoor environments, the problem of simultaneous odometry and mapping for large-scale scenarios using incremental LiDAR data remains unexplored. To bridge this gap, in this paper, we propose a novel NeRF-based LiDAR odometry and m…
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；The Hong Kong University of Science and Technology（香港科技大学）；Intel Labs China（英特尔中国研究院）；Peking University（北京大学）
Recent advances show that semi-supervised implicit representation learning can be achieved through physical constraints like Eikonal equations. However, this scheme has not yet been successfully used for LiDAR point cloud data, due to its spatially varying sparsity. In this paper, we develop a novel formulation that conditions the semi-supervised implicit function on localized shape embeddings. It exploits the strong representation learning power of sparse convolutional networks to generate shape-aware dense feature volumes, while still allows semi-supervised signed distance function learning …
Singapore University of Technology and Design（新加坡科技设计大学）；Southwest University of Science and Technology（西南科技大学）；University of Turbat（图尔巴特大学）
Environment mapping is an essential prerequisite for mobile robots to perform different tasks such as navigation and mission planning. With the availability of low-cost 2D LiDARs, there are increasing applications of such 2D LiDARs in industrial environments. However, environment mapping in an unknown and feature-less environment with such low-cost 2D LiDARs remains a challenge. The challenge mainly originates from the short-range of LiDARs and complexities in performing scan matching in these environments. In order to resolve these shortcomings, we propose to fuse the ultra-wideband (UWB) wit…
LiDAR-VGGT: Cross-Modal Coarse-to-Fine Fusion for Globally Consistent and Metric-Scale Dense Mapping
State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University（工业控制技术国家重点实验室，系统与控制研究院，浙江大学）；Differential Robot Technology Co., Ltd.（差分机器人技术有限公司）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）
Multimodal Signal Processing For Thermo-Visible-Lidar Fusion In Real-time 3D Semantic Mapping
College of Mechatronics and Control Engineering, Shenzhen University（深圳大学机械与控制工程学院）；School of Robotics, Xi’an-Jiaotong Liverpool University（西安交通大学利物浦大学机器人学院）
LISNeRF Mapping: LiDAR-based Implicit Mapping via Semantic Neural Fields for Large-Scale 3D Scenes
School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；TUM School of Computation, Information and Technology (CIT), Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）
An automated vehicle operating in an urban environment must be able to perceive and recognise object/obstacles in a three-dimensional world while navigating in a constantly changing environment. In order to plan and execute accurate sophisticated driving maneuvers, a high-level contextual understanding of the surroundings is essential. Due to the recent progress in image processing, it is now possible to obtain high definition semantic information in 2D from monocular cameras, though cameras cannot reliably provide the highly accurate 3D information provided by lasers. The fusion of these two …
Range Image-Based Implicit Neural Compression for LiDAR Point Clouds
Graduate School of Information Science and Technology（信息科学与技术研究生学校）；The University of Osaka（大阪大学）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））
Ford AV LLC（福特自动驾驶有限责任公司）；Ford Motor Company（福特汽车公司）
This paper presents a localization technique using aerial imagery maps and LIDAR based ground reflectivity for autonomous vehicles in urban environments. Traditional localization techniques using LIDAR reflectivity rely on high definition reflectivity maps generated from a mapping vehicle. The cost and effort required to maintain such prior maps are generally very high because it requires a fleet of expensive mapping vehicles. In this work we propose a localization technique where the vehicle localizes using aerial/satellite imagery, eradicating the need to develop and maintain complex high-de…
Two core competencies of a mobile robot are to build a map of the environment and to estimate its own pose on the basis of this map and incoming sensor readings. To account for the uncertainties in this process, one typically employs probabilistic state estimation approaches combined with a model of the specific sensor. Over the past years, lidar sensors have become a popular choice for mapping and localization. However, many common lidar models perform poorly in unstructured, unpredictable environments, they lack a consistent physical model for both mapping and localization, and they do not e…
University of Waterloo（滑铁卢大学）；Central University of Finance and Economics（中央财经大学）
Stack interchanges are essential components of transportation systems. Mobile laser scanning (MLS) systems have been widely used in road infrastructure mapping, but accurate mapping of complicated multi-layer stack interchanges are still challenging. This study examined the point clouds collected by a new Unmanned Aerial Vehicle (UAV) Light Detection and Ranging (LiDAR) system to perform the semantic segmentation task of a stack interchange. An end-to-end supervised 3D deep learning framework was proposed to classify the point clouds. The proposed method has proven to capture 3D features in co…
TU Delft（代尔夫特理工大学）；Mercedes-Benz AG（梅赛德斯-奔驰集团）；Esslingen University of Applied Sciences（埃斯林根应用科学大学）
Semantic scene completion is the task of jointly estimating 3D geometry and semantics of objects and surfaces within a given extent. This is a particularly challenging task on real-world data that is sparse and occluded. We propose a scene segmentation network based on local Deep Implicit Functions as a novel learning-based method for scene completion. Unlike previous work on scene completion, our method produces a continuous scene representation that is not based on voxelization. We encode raw point clouds into a latent space locally and at multiple spatial resolutions. A global scene complet…
An Efficient Probabilistic Solution to Mapping Errors in LiDAR-Camera Fusion for Autonomous Vehicles
Indiana University Purdue University Indianapolis（印第安纳大学与普渡大学印第安纳波利斯联合分校）；Purdue University（普渡大学）；Toyota Motor Engineering and Manufacturing North America Inc.（丰田汽车工程与制造北美公司）
Tongji University（同济大学）；The Key Laboratory of Embedded System and Service Computing, Ministry of Education（教育部嵌入式系统与服务计算重点实验室）；Department of Computer Science and Technology, School of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院计算机科学与技术系）；School of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院）；The Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University, Shanghai（同济大学教育部嵌入式系统与服务计算重点实验室（上海））；Department of Computer Science and Technology, School of Electronics and Information Engineering, Tongji University, Shanghai（同济大学电子与信息工程学院计算机科学与技术系（上海））
In this paper, we studied a SLAM method for vector-based road structure mapping using multi-beam LiDAR. We propose to use the polyline as the primary mapping element instead of grid cell or point cloud, because the vector-based representation is precise and lightweight, and it can directly generate vector-based High-Definition (HD) driving map as demanded by autonomous driving systems. We explored: 1) the extraction and vectorization of road structures based on local probabilistic fusion. 2) the efficient vector-based matching between frames of road structures. 3) the loop closure and optimiza…
Registration of 3D LiDAR point clouds with optical images is critical in the combination of multi-source data. Geometric misalignment originally exists in the pose data between LiDAR point clouds and optical images. To improve the accuracy of the initial pose and the applicability of the integration of 3D points and image data, we develop a simple but efficient registration method. We firstly extract point features from LiDAR point clouds and images: point features is extracted from single-frame LiDAR and point features from images using classical Canny method. Cost map is subsequently built b…
University of Bonn（波恩大学）；University of Oxford（牛津大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉玛尔机器学习与人工智能研究所）
Accurate mapping of large-scale environments is an essential building block of most outdoor autonomous systems. Challenges of traditional mapping methods include the balance between memory consumption and mapping accuracy. This paper addresses the problem of achieving large-scale 3D reconstruction using implicit representations built from 3D LiDAR measurements. We learn and store implicit features through an octree-based, hierarchical structure, which is sparse and extensible. The implicit features can be turned into signed distance values through a shallow neural network. We leverage binary c…
Mapping and 3D detection are two major issues in vision-based robotics, and self-driving. While previous works only focus on each task separately, we present an innovative and efficient multi-task deep learning framework (SM3D) for Simultaneous Mapping and 3D Detection by bridging the gap with robust depth estimation and "Pseudo-LiDAR" point cloud for the first time. The Mapping module takes consecutive monocular frames to generate depth and pose estimation. In 3D Detection module, the depth estimation is projected into 3D space to generate "Pseudo-LiDAR" point cloud, where LiDAR-based 3D dete…
LiDARs have been widely adopted to modern self-driving vehicles, providing 3D information of the scene and surrounding objects. However, adverser weather conditions still pose significant challenges to LiDARs since point clouds captured during snowfall can easily be corrupted. The resulting noisy point clouds degrade downstream tasks such as mapping. Existing works in de-noising point clouds corrupted by snow are based on nearest-neighbor search, and thus do not scale well with modern LiDARs which usually capture $100k$ or more points at 10Hz. In this paper, we introduce an unsupervised de-noi…
RAM-LAB（机器人与自动化实验室）；Hong Kong University of Science and Technology（香港科技大学）
Ego-motion estimation is a fundamental requirement for most mobile robotic applications. By sensor fusion, we can compensate the deficiencies of stand-alone sensors and provide more reliable estimations. We introduce a tightly coupled lidar-IMU fusion method in this paper. By jointly minimizing the cost derived from lidar and IMU measurements, the lidar-IMU odometry (LIO) can perform well with acceptable drift after long-term experiment, even in challenging cases where the lidar measurements can be degraded. Besides, to obtain more reliable estimations of the lidar poses, a rotation-constraine…
In this work, we present a detailed comparison of ten different 3D LiDAR sensors, covering a range of manufacturers, models, and laser configurations, for the tasks of mapping and vehicle localization, using as common reference the Normal Distributions Transform (NDT) algorithm implemented in the self-driving open source platform Autoware. LiDAR data used in this study is a subset of our LiDAR Benchmarking and Reference (LIBRE) dataset, captured independently from each sensor, from a vehicle driven on public urban roads multiple times, at different times of the day. In this study, we analyze t…
Hong Kong University of Science and Technology（香港科技大学）
Combining multiple LiDARs enables a robot to maximize its perceptual awareness of environments and obtain sufficient measurements, which is promising for simultaneous localization and mapping (SLAM). This paper proposes a system to achieve robust and simultaneous extrinsic calibration, odometry, and mapping for multiple LiDARs. Our approach starts with measurement preprocessing to extract edge and planar features from raw measurements. After a motion and extrinsic initialization procedure, a sliding window-based multi-LiDAR odometry runs onboard to estimate poses with online calibration refine…
LiDAR odometry and mapping (LOAM) has been playing an important role in autonomous vehicles, due to its ability to simultaneously localize the robot's pose and build high-precision, high-resolution maps of the surrounding environment. This enables autonomous navigation and safe path planning of autonomous vehicles. In this paper, we present a robust, real-time LOAM algorithm for LiDARs with small FoV and irregular samplings. By taking effort on both front-end and back-end, we address several fundamental challenges arising from such LiDARs, and achieve better performance in both precision and e…
University of Lincoln（林肯大学）；University of Technology of Belfort-Montbéliard（贝尔福-蒙贝利亚尔技术大学）
This paper presents a novel semantic mapping approach, Recurrent-OctoMap, learned from long-term 3D Lidar data. Most existing semantic mapping approaches focus on improving semantic understanding of single frames, rather than 3D refinement of semantic maps (i.e. fusing semantic observations). The most widely-used approach for 3D semantic map refinement is a Bayesian update, which fuses the consecutive predictive probabilities following a Markov-Chain model. Instead, we propose a learning approach to fuse the semantic features, rather than simply fusing predictions from a classifier. In our app…
The Institute of Industrial Science, The University of Tokyo（东京大学生产技术研究所）；The National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）
A unified neural network structure is presented for joint 3D object detection and point cloud segmentation in this paper. We leverage rich supervision from both detection and segmentation labels rather than using just one of them. In addition, an extension based on single-stage object detectors is proposed based on the implicit function widely used in 3D scene and object understanding. The extension branch takes the final feature map from the object detection module as input, and produces an implicit function that generates semantic distribution for each point for its corresponding voxel cente…
AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；DarkMatter AI Research（DarkMatter AI研究）；University of Oxford（牛津大学）；HKU（香港大学）；Cainiao Group（菜鸟集团）；NCUT（北方工业大学）；Sun Yat-sen University（中山大学）；Westlake University（西湖大学）
In this letter, we propose a novel LiDAR-Inertial-Visual sensor fusion framework termed R3LIVE, which takes advantage of measurement of LiDAR, inertial, and visual sensors to achieve robust and accurate state estimation. R3LIVE is contained of two subsystems, the LiDAR-inertial odometry (LIO) and visual-inertial odometry (VIO). The LIO subsystem (FAST-LIO) takes advantage of the measurement from LiDAR and inertial sensors and builds the geometry structure of (i.e. the position of 3D points) global maps. The VIO subsystem utilizes the data of visual-inertial sensors and renders the map's textur…
Multi-LiDAR Localization and Mapping Pipeline for Urban Autonomous Driving
Institute of Automotive Technology, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich（慕尼黑工业大学汽车技术研究所，慕尼黑机器人与机器智能研究所）；Professorship Autonomous Vehicle Systems, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich（慕尼黑工业大学自动驾驶车辆系统教授席，慕尼黑机器人与机器智能研究所）
University of Pennsylvania（宾夕法尼亚大学）；University of Sao Paulo（圣保罗大学）
This paper describes an end-to-end pipeline for tree diameter estimation based on semantic segmentation and lidar odometry and mapping. Accurate mapping of this type of environment is challenging since the ground and the trees are surrounded by leaves, thorns and vines, and the sensor typically experiences extreme motion. We propose a semantic feature based pose optimization that simultaneously refines the tree models while estimating the robot pose. The pipeline utilizes a custom virtual reality tool for labeling 3D scans that is used to train a semantic segmentation network. The masked point…
