We propose SampleDepth, a Convolutional Neural Network (CNN), that is suited for an adaptive LiDAR. Typically,LiDAR sampling strategy is pre-defined, constant and independent of the observed scene. Instead of letting a LiDAR sample the scene in this agnostic fashion, SampleDepth determines, adaptively, where it is best to sample the current frame. To do that, SampleDepth uses depth samples from previous time steps to predict a sampling mask for the current frame. Crucially, SampleDepth is trained to optimize the performance of a depth completion downstream task. SampleDepth is evaluated on two different depth completion networks and two LiDAR datasets, KITTI Depth Completion and the newly introduced synthetic dataset, SHIFT. We show that SampleDepth is effective and suitable for different depth completion downstream tasks.
NeuroLiDAR: Adaptive Frame Rate Depth Sensing via Neuromorphic Event-LiDAR Fusion
Singapore Management University（新加坡国立管理学院）；Singapore-MIT Alliance for Research and Technology Centre（新加坡-麻省理工联合研究和技术中心）；University of Technology Sydney（悉尼技术大学）
Learning from Mistakes: Loss-Aware Memory Enhanced Continual Learning for LiDAR Place Recognition
Shanghai Research Institute for Intelligent Autonomous System, Tongji University（上海智能自主系统研究院，同济大学）；Department of Computer Science and Technology, School of Electronics and Information Engineering, Tongji University（计算机科学与技术系，电子信息工程学院，同济大学）；MOE Key Lab of Embedded System and Service Computing, Tongji University（教育部嵌入式系统与服务计算重点实验室，同济大学）；Institute of Intelligent Vehicles, Tongji University（智能车辆研究院，同济大学）；College of Surveying and Geo-Informatics, Tongji University（测绘地理信息学院，同济大学）
Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation
East China University of Science and Technology（华东理工大学）；University of Sydney（悉尼大学）；College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学技术学院）
Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；The University of Hong Kong（香港大学）
In this technical report, we present the top-performing LiDAR-only solutions for 3D detection, 3D tracking and domain adaptation three tracks in Waymo Open Dataset Challenges 2020. Our solutions for the competition are built upon our recent proposed PV-RCNN 3D object detection framework. Several variants of our PV-RCNN are explored, including temporal information incorporation, dynamic voxelization, adaptive training sample selection, classification with RoI features, etc. A simple model ensemble strategy with non-maximum-suppression and box voting is adopted to generate the final results. By …
TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation
SAIVT Group, School of Electrical Engineering and Robotics, Queensland University of Technology（澳大利亚昆士兰科技大学电气工程与机器人学院SAIVT组）；CSIRO Robotics, CSIRO（澳大利亚联邦科学与工业研究组织机器人部）
Korea University（高丽大学）；Hyundai Motor Company（现代汽车公司）
An autonomous driving system requires a 3D object detector, which must perceive all present road agents reliably to navigate an environment safely. However, real-world driving datasets often suffer from the problem of data imbalance, which causes difficulties in training a model that works well across all classes, resulting in an undesired imbalanced sub-optimal performance. In this work, we propose a method to address this data imbalance problem. Our method consists of two main components: (i) a LiDAR-based 3D object detector with per-class multiple detection heads where losses from each head…
Adaptive Augmentation-Aware Latent Learning for Robust LiDAR Semantic Segmentation
University of Science and Technology of China（中国科学技术大学）；National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory（深空探测全国重点实验室（深空探测实验室））
Fast Attention-Based Simplification of LiDAR Point Clouds for Object Detection and Classification
Institute for Computer Science and Control (SZTAKI)（计算机科学与控制研究所）；School of Information Engineering（信息工程学院）；Faculty of Technology, Arts, and Culture（技术、艺术与文化学院）；School of Control Science and Engineering（控制科学与工程学院）
NEC Labs America（美国电气股份有限公司实验室）；University of Florida（佛罗里达大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
We present a proof-of-concept LIDAR design that allows adaptive real-time measurements according to dynamically specified measurement patterns. We describe our optical setup and calibration, which enables fast sparse depth measurements using a scanning MEMS (micro-electro-mechanical) mirror. We validate the efficacy of our prototype LIDAR design by testing on over 75 static and dynamic scenes spanning a range of environments. We show CNN-based depth-map completion experiments which demonstrate that our sensor can realize adaptive depth sensing for dynamic scenes.
University of California, Berkeley（加州大学伯克利分校）；Facebook Inc（Facebook公司）
LiDAR point-cloud segmentation is an important problem for many applications. For large-scale point cloud segmentation, the \textit{de facto} method is to project a 3D point cloud to get a 2D LiDAR image and use convolutions to process it. Despite the similarity between regular RGB and LiDAR images, we discover that the feature distribution of LiDAR images changes drastically at different image locations. Using standard convolutions to process such LiDAR images is problematic, as convolution filters pick up local features that are only active in specific regions in the image. As a result, the …
Simultaneous Localization and Mapping (SLAM) is an essential capability for autonomous robots, but due to high data rates of 3D LiDARs real-time SLAM is challenging. We propose a real-time method for 6D LiDAR odometry. Our approach combines a continuous-time B-Spline trajectory representation with a Gaussian Mixture Model (GMM) formulation to jointly align local multi-resolution surfel maps. Sparse voxel grids and permutohedral lattices ensure fast access to map surfels, and an adaptive resolution selection scheme effectively speeds up registration. A thorough experimental evaluation shows the…
Cooperative perception allows a Connected Autonomous Vehicle (CAV) to interact with the other CAVs in the vicinity to enhance perception of surrounding objects to increase safety and reliability. It can compensate for the limitations of the conventional vehicular perception such as blind spots, low resolution, and weather effects. An effective feature fusion model for the intermediate fusion methods of cooperative perception can improve feature selection and information aggregation to further enhance the perception accuracy. We propose adaptive feature fusion models with trainable feature sele…
We present an imaging framework which converts three images from a gated camera into high-resolution depth maps with depth accuracy comparable to pulsed lidar measurements. Existing scanning lidar systems achieve low spatial resolution at large ranges due to mechanically-limited angular sampling rates, restricting scene understanding tasks to close-range clusters with dense sampling. Moreover, today's pulsed lidar scanners suffer from high cost, power consumption, large form-factors, and they fail in the presence of strong backscatter. We depart from point scanning and demonstrate that it is p…
University of Electronic Science and Technology of China（电子科技大学）
LiDAR point cloud frame interpolation, which synthesizes the intermediate frame between the captured frames, has emerged as an important issue for many applications. Especially for reducing the amounts of point cloud transmission, it is by predicting the intermediate frame based on the reference frames to upsample data to high frame rate ones. However, due to high-dimensional and sparse characteristics of point clouds, it is more difficult to predict the intermediate frame for LiDAR point clouds than videos. In this paper, we propose a novel LiDAR point cloud frame interpolation method, which …
Situational awareness as a necessity in the connected and autonomous vehicles (CAV) domain is the subject of a significant number of researches in recent years. The driver's safety is directly dependent on the robustness, reliability, and scalability of such systems. Cooperative mechanisms have provided a solution to improve situational awareness by utilizing high speed wireless vehicular networks. These mechanisms mitigate problems such as occlusion and sensor range limitation. However, the network capacity is a factor determining the maximum amount of information being shared among cooperati…
University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing University of Chemical Technology（北京化工大学）；University of Science and Technology Beijing（北京科技大学）；North China University of Technology（北方工业大学）
This paper focuses on the construction of stronger local features and the effective fusion of image and LiDAR data. We adopt different modalities of LiDAR data to generate richer features and present an adaptive and azimuth-aware network to aggregate local features from image, bird's eye view maps and point cloud. Our network mainly consists of three subnetworks: ground plane estimation network, region proposal network and adaptive fusion network. The ground plane estimation network extracts features of point cloud and predicts the parameters of a plane which are used for generating abundant 3…
Recent years have witnessed the increasing application of place recognition in various environments, such as city roads, large buildings, and a mix of indoor and outdoor places. This task, however, still remains challenging due to the limitations of different sensors and the changing appearance of environments. Current works only consider the use of individual sensors, or simply combine different sensors, ignoring the fact that the importance of different sensors varies as the environment changes. In this paper, an adaptive weighting visual-LiDAR fusion method, named AdaFusion, is proposed to …
OPAL: Visibility-aware LiDAR-to-OpenStreetMap Place Recognition via Adaptive Radial Fusion
Technical University of Munich（慕尼黑技术大学）；Wuhan University（武汉大学）；University of Science and Technology of China（中国科学技术大学）；University of Cambridge（剑桥大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
Beijing Jiaotong University（北京交通大学）；Beijing Key Laboratory of Advanced Information Science and Network Technology（北京市先进信息科学与网络技术重点实验室）
LiDAR sensors are widely used in autonomous driving due to the reliable 3D spatial information. However, the data of LiDAR is sparse and the frequency of LiDAR is lower than that of cameras. To generate denser point clouds spatially and temporally, we propose the first future pseudo-LiDAR frame prediction network. Given the consecutive sparse depth maps and RGB images, we first predict a future dense depth map based on dynamic motion information coarsely. To eliminate the errors of optical flow estimation, an inter-frame aggregation module is proposed to fuse the warped depth maps with adaptiv…
Technion - Israel Institute of Technology（以色列理工学院）
This work considers the problem of depth completion, with or without image data, where an algorithm may measure the depth of a prescribed limited number of pixels. The algorithmic challenge is to choose pixel positions strategically and dynamically to maximally reduce overall depth estimation error. This setting is realized in daytime or nighttime depth completion for autonomous vehicles with a programmable LiDAR. Our method uses an ensemble of predictors to define a sampling probability over pixels. This probability is proportional to the variance of the predictions of ensemble members, thus …
Most learning-based methods estimate ego-motion by utilizing visual sensors, which suffer from dramatic lighting variations and textureless scenarios. In this paper, we incorporate sparse but accurate depth measurements obtained from lidars to overcome the limitation of visual methods. To this end, we design a self-supervised visual-lidar odometry (Self-VLO) framework. It takes both monocular images and sparse depth maps projected from 3D lidar points as input, and produces pose and depth estimations in an end-to-end learning manner, without using any ground truth labels. To effectively fuse t…
LiDAR-based 3D object detection and panoptic segmentation are two crucial tasks in the perception systems of autonomous vehicles and robots. In this paper, we propose All-in-One Perception Network (AOP-Net), a LiDAR-based multi-task framework that combines 3D object detection and panoptic segmentation. In this method, a dual-task 3D backbone is developed to extract both panoptic- and detection-level features from the input LiDAR point cloud. Also, a new 2D backbone that intertwines Multi-Layer Perceptron (MLP) and convolution layers is designed to further improve the detection task performance…
