LiDAR mapping is important yet challenging in self-driving and mobile robotics. To tackle such a global point cloud registration problem, DeepMapping converts the complex map estimation into a self-supervised training of simple deep networks. Despite its broad convergence range on small datasets, DeepMapping still cannot produce satisfactory results on large-scale datasets with thousands of frames. This is due to the lack of loop closures and exact cross-frame point correspondences, and the slow convergence of its global localization network. We propose DeepMapping2 by adding two novel techniques to address these issues: (1) organization of training batch based on map topology from loop closing, and (2) self-supervised local-to-global point consistency loss leveraging pairwise registration. Our experiments and ablation studies on public datasets (KITTI, NCLT, and Nebula) demonstrate the effectiveness of our method.
Beihang University（北京航空航天大学）；National University of Singapore（新加坡国立大学）；Institute for Infocomm Research（资讯通信研究院）
Deep stereo matching has made significant progress in recent years. However, state-of-the-art methods are based on expensive 4D cost volume, which limits their use in real-world applications. To address this issue, 3D correlation maps and iterative disparity updates have been proposed. Regarding that in real-world platforms, such as self-driving cars and robots, the Lidar is usually installed. Thus we further introduce the sparse Lidar point into the iterative updates, which alleviates the burden of network updating the disparity from zero states. Furthermore, we propose training the network i…
HiLoTs: High-Low Temporal Sensitive Representation Learning for Semi-Supervised LiDAR Segmentation in Autonomous Driving
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Peking University（北京大学）；China North Vehicle Research Institute（中国北方车辆研究所）
We propose a method for off-road drivable area extraction using 3D LiDAR data with the goal of autonomous driving application. A specific deep learning framework is designed to deal with the ambiguous area, which is one of the main challenges in the off-road environment. To reduce the considerable demand for human-annotated data for network training, we utilize the information from vast quantities of vehicle paths and auto-generated obstacle labels. Using these autogenerated annotations, the proposed network can be trained using weakly supervised or semi-supervised methods, which can achieve b…
S-BEVLoc: BEV-based Self-supervised Framework for Large-scale LiDAR Global Localization
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Ningbo Innovation Center, Zhejiang University（浙江大学宁波创新中心）；College of Information Engineering, China Jiliang University（中国浙江师范大学信息工程学院）；Jinhua Institute of Zhejiang University（浙江大学金华研究院）
Temporal Overlapping Prediction: A Self-supervised Pre-training Method for LiDAR Moving Object Segmentation
The University of Hong Kong（香港大学）；KTH Royal Institute of Technology（皇家理工学院）；Southern University of Science and Technology（南方科技大学）；Shanghai AI Laboratory（上海人工智能实验室）
Beijing Jiaotong University（北京交通大学）；Beijing Key Laboratory of Advanced Information Science and Network Technology（北京市先进信息科学与网络技术重点实验室）
LiDAR sensors are widely used in autonomous driving due to the reliable 3D spatial information. However, the data of LiDAR is sparse and the frequency of LiDAR is lower than that of cameras. To generate denser point clouds spatially and temporally, we propose the first future pseudo-LiDAR frame prediction network. Given the consecutive sparse depth maps and RGB images, we first predict a future dense depth map based on dynamic motion information coarsely. To eliminate the errors of optical flow estimation, an inter-frame aggregation module is proposed to fuse the warped depth maps with adaptiv…
LIGM（法国巴黎东区大学信息与数学实验室）；Ecole des Ponts（巴黎路桥学院）；Univ. Gustave Eiffel（古斯塔夫·埃菲尔大学）；CNRS（法国国家科学研究中心）
Segmenting or detecting objects in sparse Lidar point clouds are two important tasks in autonomous driving to allow a vehicle to act safely in its 3D environment. The best performing methods in 3D semantic segmentation or object detection rely on a large amount of annotated data. Yet annotating 3D Lidar data for these tasks is tedious and costly. In this context, we propose a self-supervised pre-training method for 3D perception models that is tailored to autonomous driving data. Specifically, we leverage the availability of synchronized and calibrated image and Lidar sensors in autonomous dri…
In the era of autonomous driving, urban mapping represents a core step to let vehicles interact with the urban context. Successful mapping algorithms have been proposed in the last decade building the map leveraging on data from a single sensor. The focus of the system presented in this paper is twofold: the joint estimation of a 3D map from lidar data and images, based on a 3D mesh, and its texturing. Indeed, even if most surveying vehicles for mapping are endowed by cameras and lidar, existing mapping algorithms usually rely on either images or lidar data; moreover both image-based and lidar…
Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) to dense d…
DST-Calib: A Dual-Path, Self-Supervised, Target-Free LiDAR-Camera Extrinsic Calibration Network
Department of Control Science & Engineering, the College of Electronics & Information Engineering, Tongji University（控制科学与工程系，电子与信息工程学院，同济大学）；Department of Mechanical Engineering, the School of Mechanical Engineering, Tongji University（机械工程系，机械工程学院，同济大学）；School of Robotics, Hunan University（机器人学院，湖南大学）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科学技术大学）；Department of Control Science & Engineering, the College of Electronics & Information Engineering, Shanghai Research Institute for Intelligent Autonomous Systems, the State Key Laboratory of Intelligent Autonomous Systems, and Frontiers Science Center for Intelligent Autonomous Systems, Tongji University（控制科学与工程系，电子与信息工程学院，上海智能自主系统研究院，智能自主系统国家重点实验室，智能自主系统前沿科学中心，同济大学）；National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）
In this work, we present a detailed comparison of ten different 3D LiDAR sensors, covering a range of manufacturers, models, and laser configurations, for the tasks of mapping and vehicle localization, using as common reference the Normal Distributions Transform (NDT) algorithm implemented in the self-driving open source platform Autoware. LiDAR data used in this study is a subset of our LiDAR Benchmarking and Reference (LIBRE) dataset, captured independently from each sensor, from a vehicle driven on public urban roads multiple times, at different times of the day. In this study, we analyze t…
As the autonomous driving industry is slowly maturing, visual map localization is quickly becoming the standard approach to localize cars as accurately as possible. Owing to the rich data returned by visual sensors such as cameras or LiDARs, researchers are able to build different types of maps with various levels of details, and use them to achieve high levels of vehicle localization accuracy and stability in urban environments. Contrary to the popular SLAM approaches, visual map localization relies on pre-built maps, and is focused solely on improving the localization accuracy by avoiding er…
In this paper we propose a real-time, calibration-agnostic and effective localization system for self-driving cars. Our method learns to embed the online LiDAR sweeps and intensity map into a joint deep embedding space. Localization is then conducted through an efficient convolutional matching between the embeddings. Our full system can operate in real-time at 15Hz while achieving centimeter level accuracy across different LiDAR sensors and environments. Our experiments illustrate the performance of the proposed approach over a large-scale dataset consisting of over 4000km of driving.
Università degli Studi di Milano - Bicocca（米兰比可卡大学）；University of Freiburg（弗莱堡大学）；Universidad de Alcalá（阿尔卡拉大学）
Global localization is an important and widely studied problem for many robotic applications. Place recognition approaches can be exploited to solve this task, e.g., in the autonomous driving field. While most vision-based approaches match an image w.r.t. an image database, global visual localization within LiDAR-maps remains fairly unexplored, even though the path toward high definition 3D maps, produced mainly from LiDARs, is clear. In this work we leverage Deep Neural Network (DNN) approaches to create a shared embedding space between images and LiDAR-maps, allowing for image to 3D-LiDAR pl…
ETH Zurich（苏黎世联邦理工学院）；MPI for Informatics（马克斯·普朗克信息学研究所）；KU Leuven（鲁汶大学）
Densely annotating LiDAR point clouds remains too expensive and time-consuming to keep up with the ever growing volume of data. While current literature focuses on fully-supervised performance, developing efficient methods that take advantage of realistic weak supervision have yet to be explored. In this paper, we propose using scribbles to annotate LiDAR point clouds and release ScribbleKITTI, the first scribble-annotated dataset for LiDAR semantic segmentation. Furthermore, we present a pipeline to reduce the performance gap that arises when using such weak annotations. Our pipeline comprise…
Most learning-based methods estimate ego-motion by utilizing visual sensors, which suffer from dramatic lighting variations and textureless scenarios. In this paper, we incorporate sparse but accurate depth measurements obtained from lidars to overcome the limitation of visual methods. To this end, we design a self-supervised visual-lidar odometry (Self-VLO) framework. It takes both monocular images and sparse depth maps projected from 3D lidar points as input, and produces pose and depth estimations in an end-to-end learning manner, without using any ground truth labels. To effectively fuse t…
Technical University of Munich（慕尼黑工业大学）；Tongji University（同济大学）
State-of-the-art lidar-based 3D object detection methods rely on supervised learning and large labeled datasets. However, annotating lidar data is resource-consuming, and depending only on supervised learning limits the applicability of trained models. Self-supervised training strategies can alleviate these issues by learning a general point cloud backbone model for downstream 3D vision tasks. Against this backdrop, we show the relationship between self-supervised multi-frame flow representations and single-frame 3D detection hypotheses. Our main contribution leverages learned flow and motion …
Seoul National University（首尔国立大学）；Agency for Defense Development（国防发展局）；Inha University（仁荷大学）
LiDAR is widely used to capture accurate 3D outdoor scene structures. However, LiDAR produces many undesirable noise points in snowy weather, which hamper analyzing meaningful 3D scene structures. Semantic segmentation with snow labels would be a straightforward solution for removing them, but it requires laborious point-wise annotation. To address this problem, we propose a novel self-supervised learning framework for snow points removal in LiDAR point clouds. Our method exploits the structural characteristic of the noise points: low spatial correlation with their neighbors. Our method consis…
Collaborative Learning for Semi-Supervised LiDAR Semantic Segmentation
Bosch Research, Robert Bosch GmbH, Stuttgart, Germany（博世研究, 博世有限公司, 斯图加特, 德国）；Institute for Neuro- and Bioinformatics, University of Lübeck, Lübeck, Germany（神经与生物信息学研究所, 奥尔德马克大学, 奥尔德马克, 德国）
Perception and localization are essential for autonomous delivery vehicles, mostly estimated from 3D LiDAR sensors due to their precise distance measurement capability. This paper presents a strategy to obtain the real-time pseudo point cloud instead of the laser sensor from the image sensor. We propose an approach to use different depth estimators to obtain pseudo point clouds like LiDAR to obtain better performance. Moreover, the training and validating strategy of the depth estimator has adopted stereo imagery data to estimate more accurate depth estimation as well as point cloud results. O…
Chalmers University of Technology（查尔姆斯理工大学）；AB Volvo（沃尔沃集团）
Recent advances in the field of machine learning and computer vision have enabled the development of fast and accurate road detectors. Commonly such systems are trained within a supervised learning paradigm where both an input sensor's data and the corresponding ground truth label must be provided. The task of generating labels is commonly carried out by human annotators and it is notoriously time consuming and expensive. In this work, it is shown that a semi-supervised approach known as co-training can provide significant F1-score average improvements compared to supervised learning. In co-tr…
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；The Hong Kong University of Science and Technology（香港科技大学）；Intel Labs China（英特尔中国研究院）；Peking University（北京大学）
Recent advances show that semi-supervised implicit representation learning can be achieved through physical constraints like Eikonal equations. However, this scheme has not yet been successfully used for LiDAR point cloud data, due to its spatially varying sparsity. In this paper, we develop a novel formulation that conditions the semi-supervised implicit function on localized shape embeddings. It exploits the strong representation learning power of sparse convolutional networks to generate shape-aware dense feature volumes, while still allows semi-supervised signed distance function learning …
ItTakesTwo: Leveraging Peer Representations for Semi-supervised LiDAR Semantic Segmentation
Australian Institute for Machine Learning (AIML), University of Adelaide（澳大利亚机器学习研究所（AIML）、阿德莱德大学）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；University of Erlangen–Nuremberg（埃尔兰根-纽伦堡大学）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK（视觉、语音和信号处理中心（CVSSP）、萨里大学）
Poles and building edges are frequently observable objects on urban roads, conveying reliable hints for various computer vision tasks. To repetitively extract them as features and perform association between discrete LiDAR frames for registration, we propose the first learning-based feature segmentation and description model for 3D lines in LiDAR point cloud. To train our model without the time consuming and tedious data labeling process, we first generate synthetic primitives for the basic appearance of target lines, and build an iterative line auto-labeling process to gradually refine line l…
Clemson University（克莱姆森大学）；The City University of New York（纽约市立大学）；Carnegie Mellon University（卡内基梅隆大学）
Self-supervised monocular depth prediction provides a cost-effective solution to obtain the 3D location of each pixel. However, the existing approaches usually lead to unsatisfactory accuracy, which is critical for autonomous robots. In this paper, we propose FusionDepth, a novel two-stage network to advance the self-supervised monocular dense depth learning by leveraging low-cost sparse (e.g. 4-beam) LiDAR. Unlike the existing methods that use sparse LiDAR mainly in a manner of time-consuming iterative post-processing, our model fuses monocular image features and sparse LiDAR features to pred…
Arizona State University（亚利桑那州立大学）；U.S. Army DEVCOM Research Laboratory（美国陆军作战能力发展司令部研究实验室）；U.S. Army DEVCOM C5ISR Center（美国陆军作战能力发展司令部指挥、控制、通信、计算机、网络、情报、监视与侦察中心）
Motivated by the increasing application of low-resolution LiDAR recently, we target the problem of low-resolution LiDAR-camera calibration in this work. The main challenges are two-fold: sparsity and noise in point clouds. To address the problem, we propose to apply depth interpolation to increase the point density and supervised contrastive learning to learn noise-resistant features. The experiments on RELLIS-3D demonstrate that our approach achieves an average mean absolute rotation/translation errors of 0.15cm/0.33\textdegree on 32-channel LiDAR point cloud data, which significantly outperf…
Samsung Electronics Co., Ltd.（三星电子有限公司）；Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术院）
There have been attempts to detect 3D objects by fusion of stereo camera images and LiDAR sensor data or using LiDAR for pre-training and only monocular images for testing, but there have been less attempts to use only monocular image sequences due to low accuracy. In addition, when depth prediction using only monocular images, only scale-inconsistent depth can be predicted, which is the reason why researchers are reluctant to use monocular images alone. Therefore, we propose a method for predicting absolute depth and detecting 3D objects using only monocular image sequences by enabling end-to…
ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Depth estimation is usually ill-posed and ambiguous for monocular camera-based 3D multi-person pose estimation. Since LiDAR can capture accurate depth information in long-range scenes, it can benefit both the global localization of individuals and the 3D pose estimation by providing rich geometry features. Motivated by this, we propose a monocular camera and single LiDAR-based method for 3D multi-person pose estimation in large-scale scenes, which is easy to deploy and insensitive to light. Specifically, we design an effective fusion strategy to take advantage of multi-modal input data, includ…
Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration
School of Mechatronic Engineering and Automation, Shanghai University（机械电子工程与自动化学院，上海大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，城市大学）；School of Artificial Intelligence, Xidian University（人工智能学院，西安电子科技大学）；Department of Electronic Engineering, The Chinese University of Hong Kong（电子工程系，香港中文大学）
ContrastMotion: Self-supervised Scene Motion Learning for Large-Scale LiDAR Point Clouds
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Rhino（犀牛（公司））；The Chinese University of Hong Kong（香港中文大学）；OPPO Research Institute（OPPO研究院）；Zhejiang Lab（之江实验室）；ShanghaiTech University（上海科技大学）
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）
This paper introduces the Masked Voxel Jigsaw and Reconstruction (MV-JAR) method for LiDAR-based self-supervised pre-training and a carefully designed data-efficient 3D object detection benchmark on the Waymo dataset. Inspired by the scene-voxel-point hierarchy in downstream 3D object detectors, we design masking and reconstruction strategies accounting for voxel distributions in the scene and local point distributions within the voxel. We employ a Reversed-Furthest-Voxel-Sampling strategy to address the uneven distribution of LiDAR points and propose MV-JAR, which combines two techniques for …
LASTIG, Univ. Gustave Eiffel, IGN-ENSG（古斯塔夫·埃菲尔大学LASTIG实验室，法国国家地理与林业信息学院）；INRAE, UMR 1202 BIOGECO, Université de Bordeaux（法国国家农业食品与环境研究院，波尔多大学生物地球化学与生态学联合研究中心）；EA G&E Bordeaux INP, Université Bordeaux Montaigne（波尔多蒙田大学，波尔多国立高等电子、计算机、电信、数学与机械学院环境与地理空间工程研究团队）
We propose a new deep learning-based method for estimating the occupancy of vegetation strata from airborne 3D LiDAR point clouds. Our model predicts rasterized occupancy maps for three vegetation strata corresponding to lower, medium, and higher cover. Our weakly-supervised training scheme allows our network to only be supervised with vegetation occupancy values aggregated over cylindrical plots containing thousands of points. Such ground truth is easier to produce than pixel-wise or point-wise annotations. Our method outperforms handcrafted and deep learning baselines in terms of precision b…
Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；SenseTime Research（商汤科技研究院）
Recent learning-based LiDAR odometry methods have demonstrated their competitiveness. However, most methods still face two substantial challenges: 1) the 2D projection representation of LiDAR data cannot effectively encode 3D structures from the point clouds; 2) the needs for a large amount of labeled data for training limit the application scope of these methods. In this paper, we propose a self-supervised LiDAR odometry method, dubbed SelfVoxeLO, to tackle these two difficulties. Specifically, we propose a 3D convolution network to process the raw LiDAR data directly, which extracts features…
