Semantic2D: Enabling Semantic Scene Understanding with 2D Lidar Alone
School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Department of Mechanical Engineering, Temple University（机械工程系， Temple大学）
Zhanteng Xie, Yipeng Pan, Yinqiang Zhang, Jia Pan, Philip Dames
This article presents a complete semantic scene understanding workflow using only a single 2D lidar. This fills the gap in 2D lidar semantic segmentation, thereby enabling the rethinking and enhancement of existing 2D lidar-based algorithms for application in various mobile robot tasks. It introduces the first publicly available 2D lidar semantic segmentation dataset and the first fine-grained semantic segmentation algorithm specifically designed for 2D lidar sensors on autonomous mobile robots. To annotate this dataset, we propose a novel semi-automatic semantic labeling framework that requires minimal human effort and provides point-level semantic annotations. The data was collected by three different types of 2D lidar sensors across twelve indoor environments, featuring a range of common indoor objects. Furthermore, the proposed semantic segmentation algorithm fully exploits raw lidar information -- position, range, intensity, and incident angle -- to deliver stochastic, point-wise semantic segmentation. We present a series of semantic occupancy grid mapping experiments and demonstrate two semantically-aware navigation control policies based on 2D lidar. These results demonstrate that the proposed semantic 2D lidar dataset, semi-automatic labeling framework, and segmentation algorithm are effective and can enhance different components of the robotic navigation pipeline. Multimedia resources are available at: https://youtu.be/P1Hsvj6WUSY.
Peking University（北京大学）；China North Vehicle Research Institute（中国北方车辆研究所）
This work studies the semantic segmentation of 3D LiDAR data in dynamic scenes for autonomous driving applications. A system of semantic segmentation using 3D LiDAR data, including range image segmentation, sample generation, inter-frame data association, track-level annotation and semi-supervised learning, is developed. To reduce the considerable requirement of fine annotations, a CNN-based classifier is trained by considering both supervised samples with manually labeled object classes and pairwise constraints, where a data sample is composed of a segment as the foreground and neighborhood p…
For autonomous vehicles, an accurate calibration for LiDAR and camera is a prerequisite for multi-sensor perception systems. However, existing calibration techniques require either a complicated setting with various calibration targets, or an initial calibration provided beforehand, which greatly impedes their applicability in large-scale autonomous vehicle deployment. To tackle these issues, we propose a novel method to calibrate the extrinsic parameter for LiDAR and camera in road scenes. Our method introduces line features from static straight-line-shaped objects such as road lanes and pole…
Real-Time Spatial Reasoning by Mobile Robots for Reconstruction and Navigation in Dynamic LiDAR Scenes
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science, University of Guelph（圭尔夫大学计算机科学学院）；School of Computing Science, Simon Fraser University（西蒙弗雷泽大学计算机科学学院）
FlowCalib: LiDAR-to-Vehicle Miscalibration Detection using Scene Flows
TUM School of Engineering and Design, Chair of Automotive Technology, Technical University of Munich（慕尼黑技术大学工程与设计学院，汽车技术教授职位）；TUM School of Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）
Feasibility of Indoor Frame-Wise Lidar Semantic Segmentation via Distillation from Visual Foundation Model
1 Department of Earth Observation Science, Faculty of Geo-Information Science；Earth Observation (ITC), University of Twente, 7522 NB Enschede, The Netherlands
Kyushu University（九州大学）；Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）
3D LiDAR sensors are indispensable for the robust vision of autonomous mobile robots. However, deploying LiDAR-based perception algorithms often fails due to a domain gap from the training environment, such as inconsistent angular resolution and missing properties. Existing studies have tackled the issue by learning inter-domain mapping, while the transferability is constrained by the training configuration and the training is susceptible to peculiar lossy noises called ray-drop. To address the issue, this paper proposes a generative model of LiDAR range images applicable to the data-level dom…
Object detection and motion parameters estimation are crucial tasks for self-driving vehicle safe navigation in a complex urban environment. In this work we propose a novel real-time approach of temporal context aggregation for motion detection and motion parameters estimation based on 3D point cloud sequence. We introduce an ego-motion compensation layer to achieve real-time inference with performance comparable to a naive odometric transform of the original point cloud sequence. Not only is the proposed architecture capable of estimating the motion of common road participants like vehicles o…
DiffSSC: Semantic LiDAR Scan Completion using Denoising Diffusion Probabilistic Models
Autonomous Intelligent Systems group, Computer Science Institute VI – Intelligent Systems and Robotics – and the Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany（自主智能系统组，计算机科学研究所VI——智能系统与机器人——和机器人中心以及拉马尔人工智能与机器学习研究所，波恩大学，德国）
UC San Diego（加州大学圣地亚哥分校）；Waymo（Waymo公司）；Google（谷歌公司）
Semantic segmentation of LiDAR point clouds is an important task in autonomous driving. However, training deep models via conventional supervised methods requires large datasets which are costly to label. It is critical to have label-efficient segmentation approaches to scale up the model to new operational domains or to improve performance on rare cases. While most prior works focus on indoor scenes, we are one of the first to propose a label-efficient semantic segmentation pipeline for outdoor scenes with LiDAR point clouds. Our method co-designs an efficient labeling process with semi/weakl…
Semantic grids are a useful representation of the environment around a robot. They can be used in autonomous vehicles to concisely represent the scene around the car, capturing vital information for downstream tasks like navigation or collision assessment. Information from different sensors can be used to generate these grids. Some methods rely only on RGB images, whereas others choose to incorporate information from other sensors, such as radar or LiDAR. In this paper, we present an architecture that fuses LiDAR and camera information to generate semantic grids. By using the 3D information fr…
We propose a graph neural network(GNN) based method to incorporate scene context for the semantic segmentation of 3D LiDAR data. The problem is defined as building a graph to represent the topology of a center segment with its neighborhoods, then inferring the segment label. The node of graph is generated from the segment on range image, which is suitable for both sparse and dense point cloud. Edge weights that evaluate the correlations of center node and its neighborhoods are automatically encoded by a neural network, therefore the number of neighbor nodes is no longer a sensitive parameter. …
Localization is an essential technique in mobile robotics. In a complex environment, it is necessary to fuse different localization modules to obtain more robust results, in which the error model plays a paramount role. However, exteroceptive sensor-based odometries (ESOs), such as LiDAR/visual odometry, often deliver results with scene-related error, which is difficult to model accurately. To address this problem, this research designs a scene-aware error model for ESO, based on which a multimodal localization fusion framework is developed. In addition, an end-to-end learning method is propos…
Wuhan University（武汉大学）；School of Computer Science（计算机学院）；School of Electronic Information（电子信息学院）；School of Artificial Intelligence（人工智能学院）；National Engineering Research Center for Multimedia Software（多媒体软件国家工程研究中心）；Institute for Math & AI（数学与人工智能研究所）
DFKI – German Research Center for Artificial Intelligence（德国人工智能研究中心）；BMW Group（宝马集团）
We propose a new approach called LiDAR-Flow to robustly estimate a dense scene flow by fusing a sparse LiDAR with stereo images. We take the advantage of the high accuracy of LiDAR to resolve the lack of information in some regions of stereo images due to textureless objects, shadows, ill-conditioned light environment and many more. Additionally, this fusion can overcome the difficulty of matching unstructured 3D points between LiDAR-only scans. Our LiDAR-Flow approach consists of three main steps; each of them exploits LiDAR measurements. First, we build strong seeds from LiDAR to enhance the…
AEOS: Active Environment-aware Optimal Scanning Control for UAV LiDAR-Inertial Odometry in Complex Scenes
School of Electrical and Electronic Engineering（电子工程学院）；Nanyang Technological University（南洋理工大学）；Robotics Institute（机器人研究所）；Carnegie Mellon University（卡内基梅隆大学）
University of Hong Kong（香港大学）；Harbin Institute of Technology（哈尔滨工业大学）
Recent advances in LiDAR technology have opened up new possibilities for robotic navigation. Given the widespread use of occupancy grid maps (OGMs) in robotic motion planning, this paper aims to address the challenges of integrating LiDAR with OGMs. To this end, we propose ROG-Map, a uniform grid-based OGM that maintains a local map moving along with the robot to enable efficient map operation and reduce memory costs for large-scene autonomous flight. Moreover, we present a novel incremental obstacle inflation method that significantly reduces the computational cost of inflation. The proposed …
The University of Hong Kong（香港大学）；Southern University of Science and Technology（南方科技大学）
In this letter, we present a novel method for automatic extrinsic calibration of high-resolution LiDARs and RGB cameras in targetless environments. Our approach does not require checkerboards but can achieve pixel-level accuracy by aligning natural edge features in the two sensors. On the theory level, we analyze the constraints imposed by edge features and the sensitivity of calibration accuracy with respect to edge distribution in the scene. On the implementation level, we carefully investigate the physical measuring principles of LiDARs and propose an efficient and accurate LiDAR edge extra…
Semantic scene understanding is important for various applications. In particular, self-driving cars need a fine-grained understanding of the surfaces and objects in their vicinity. Light detection and ranging (LiDAR) provides precise geometric information about the environment and is thus a part of the sensor suites of almost all self-driving cars. Despite the relevance of semantic scene understanding for this application, there is a lack of a large dataset for this task which is based on an automotive LiDAR. In this paper, we introduce a large dataset to propel research on laser-based semant…
LIV-GaussMap: LiDAR-Inertial-Visual Fusion for Real-time 3D Radiance Field Map Rendering
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The University of Hong Kong（香港大学）
German Research Center for Artificial Intelligence - DFKI（德国人工智能研究中心）；Birla Institute of Technology and Science - BITS Pilani（比尔拉技术与科学学院）；University of Applied Sciences Mannheim（曼海姆应用科学大学）；University of Kaiserslautern - TUK（凯泽斯劳滕大学）
Scene flow is the dense 3D reconstruction of motion and geometry of a scene. Most state-of-the-art methods use a pair of stereo images as input for full scene reconstruction. These methods depend a lot on the quality of the RGB images and perform poorly in regions with reflective objects, shadows, ill-conditioned light environment and so on. LiDAR measurements are much less sensitive to the aforementioned conditions but LiDAR features are in general unsuitable for matching tasks due to their sparse nature. Hence, using both LiDAR and RGB can potentially overcome the individual disadvantages of…
LI-GS: Gaussian Splatting with LiDAR Incorporated for Accurate Large-Scale Reconstruction
Zhejiang University（浙江大学）；College of Control Science and Engineering（控制科学与工程学院）；State Key Laboratory of Industrial Control Technology（工业控制技术国家重点实验室）；Key Laboratory of Collaborative sensing and autonomous unmanned systems of Zhejiang Province（浙江省协同感知与自主无人系统重点实验室）
University of Toronto Institute for Aerospace Studies (UTIAS)（多伦多大学航空航天研究所）；ETH Zürich（苏黎世联邦理工学院）；Apple Inc.（苹果公司）
We present a novel method for generating, predicting, and using Spatiotemporal Occupancy Grid Maps (SOGM), which embed future information of dynamic scenes. Our automated generation process creates groundtruth SOGMs from previous navigation data. We build on prior work to annotate lidar points based on their dynamic properties, which are then projected on time-stamped 2D grids: SOGMs. We design a 3D-2D feedforward architecture, trained to predict the future time steps of SOGMs, given 3D lidar frames as input. Our pipeline is entirely self-supervised, thus enabling lifelong learning for robots.…
Detecting moving objects in dynamic scenes from sequences of lidar scans is an important task in object tracking, mapping, localization, and navigation. Many works focus on changes detection in previously observed scenes, while a very limited amount of literature addresses moving objects detection. The state-of-the-art method exploits Dempster-Shafer Theory to evaluate the occupancy of a lidar scan and to discriminate points belonging to the static scene from moving ones. In this paper we improve both speed and accuracy of this method by discretizing the occupancy representation, and by removi…
Australian Centre for Field Robotics（澳大利亚野外机器人中心）；University of Sydney（悉尼大学）；Baraja Pty Ltd（巴拉哈私人有限公司）
We propose a robust calibration pipeline that optimises the selection of calibration samples for the estimation of calibration parameters that fit the entire scene. We minimise user error by automating the data selection process according to a metric, called Variability of Quality (VOQ) that gives a score to each calibration set of samples. We show that this VOQ score is correlated with the estimated calibration parameter's ability to generalise well to the entire scene, thereby overcoming the overfitting problems of existing calibration algorithms. Our approach has the benefits of simplifying…
SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net
University of Bonn（波恩大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；Center for Robotics（机器人中心）；Computer Science Institute VI – Intelligent Systems and Robotics（计算机科学研究所VI——智能系统与机器人）
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；The University of Hong Kong（香港大学）；Hong Kong University of Science and Technology（香港科技大学）；DJI Co.（大疆创新科技有限公司）
Modeling scene geometry using implicit neural representation has revealed its advantages in accuracy, flexibility, and low memory usage. Previous approaches have demonstrated impressive results using color or depth images but still have difficulty handling poor light conditions and large-scale scenes. Methods taking global point cloud as input require accurate registration and ground truth coordinate labels, which limits their application scenarios. In this paper, we propose a new method that uses sparse LiDAR point clouds and rough odometry to reconstruct fine-grained implicit occupancy field…
Oxford Robotics Institute (ORI)（牛津机器人研究所）；University of Oxford（牛津大学）
Road boundaries, or curbs, provide autonomous vehicles with essential information when interpreting road scenes and generating behaviour plans. Although curbs convey important information, they are difficult to detect in complex urban environments (in particular in comparison to other elements of the road such as traffic signs and road markings). These difficulties arise from occlusions by other traffic participants as well as changing lighting and/or weather conditions. Moreover, road boundaries have various shapes, colours and structures while motion planning algorithms require accurate and …
LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；National University of Singapore（新加坡国立大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University of Posts and Telecommunications（南京邮电大学）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）
