Autonomous driving in urban crowds at unregulated intersections is challenging, where dynamic occlusions and uncertain behaviors of other vehicles should be carefully considered. Traditional methods are heuristic and based on hand-engineered rules and parameters, but scale poorly in new situations. Therefore, they require high labor cost to design and maintain rules in all foreseeable scenarios. Recently, deep reinforcement learning (DRL) has shown promising results in urban driving scenarios. However, DRL is known to be sample inefficient, and most previous works assume perfect observations such as ground-truth locations and motions of vehicles without considering noises and occlusions, which might be a too strong assumption for policy deployment. In this work, we use DRL to train lidar-based end-to-end driving policies that naturally consider imperfect partial observations. We further use unsupervised contrastive representation learning as an auxiliary task to improve the sample efficiency. The comparative evaluation results reveal that our method achieves higher success rates than the state-of-the-art (SOTA) lidar-based end-to-end driving network, better trades off safety and efficiency than the carefully tuned rule-based method, and generalizes better to new scenarios than the baselines. Demo videos are available at https://caipeide.github.io/carl-lead/.
Radar and lidar, provided by two different range sensors, each has pros and cons of various perception tasks on mobile robots or autonomous driving. In this paper, a Monte Carlo system is used to localize the robot with a rotating radar sensor on 2D lidar maps. We first train a conditional generative adversarial network to transfer raw radar data to lidar data, and achieve reliable radar points from generator. Then an efficient radar odometry is included in the Monte Carlo system. Combining the initial guess from odometry, a measurement model is proposed to match the radar data and prior lidar…
4D Contrastive Superflows are Dense 3D Representation Learners
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab（注：S-Lab为南洋理工大学下属实验室，按要求提取到大学/研究院级别，此处保留原文机构名及对应翻译））
CO^3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving
The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；Huawei Noah’s Ark Lab（华为诺亚实验室）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）
Localization is a crucial capability for mobile robots and autonomous cars. In this paper, we address learning an observation model for Monte-Carlo localization using 3D LiDAR data. We propose a novel, neural network-based observation model that computes the expected overlap of two 3D LiDAR scans. The model predicts the overlap and yaw angle offset between the current sensor reading and virtual frames generated from a pre-built map. We integrate this observation model into a Monte-Carlo localization framework and tested it on urban datasets collected with a car in different seasons. The experi…
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）
This paper presents a model-free, setting-independent method for online detection of dynamic objects in 3D lidar data. We explicitly compensate for the moving-while-scanning operation (motion distortion) of present-day 3D spinning lidar sensors. Our detection method uses a motion-compensated freespace querying algorithm and classifies between dynamic (currently moving) and static (currently stationary) labels at the point level. For a quantitative analysis, we establish a benchmark with motion-distorted lidar data using CARLA, an open-source simulator for autonomous driving research. We also p…
Technical University of Munich（慕尼黑工业大学）；Sun Yat-sen University（中山大学）
Localization in the environment is an essential navigational capability for animals and mobile robots. In the indoor environment, the global localization problem remains challenging to be perfectly solved with probabilistic methods. However, animals are able to instinctively localize themselves with much less effort. Therefore, it is intriguing and promising to seek biological inspiration from animals. In this paper, we present a biologically-inspired global localization system using a LiDAR sensor that utilizes a hippocampal model and a landmark-based re-localization approach. The experiment …
Robust and accurate, map-based localization is crucial for autonomous mobile systems. In this paper, we exploit range images generated from 3D LiDAR scans to address the problem of localizing mobile robots or autonomous cars in a map of a large-scale outdoor environment represented by a triangular mesh. We use the Poisson surface reconstruction to generate the mesh-based map representation. Based on the range images generated from the current LiDAR scan and the synthetic rendered views from the mesh-based map, we propose a new observation model and integrate it into a Monte Carlo localization …
Carnegie Mellon University（卡内基梅隆大学）；Amazon（亚马逊）；University of Southern California（南加州大学）
The past few years have witnessed an increasing interest in improving the perception performance of LiDARs on autonomous vehicles. While most of the existing works focus on developing new deep learning algorithms or model architectures, we study the problem from the physical design perspective, i.e., how different placements of multiple LiDARs influence the learning-based perception. To this end, we introduce an easy-to-compute information-theoretic surrogate metric to quantitatively and fast evaluate LiDAR placement for 3D detection of different types of objects. We also present a new data co…
Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretraining
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Computing, Department of Computer Science, National University of Singapore（新加坡国立大学计算机学院）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机学院）；Shanghai AI Laboratory（上海人工智能实验室）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
Interpretable Contrastive Monte Carlo Tree Search Reasoning
The University of Sydney（悉尼大学）；Peking University（北京大学）；Xiaohongshu Inc.（小红书公司）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Peking University（北京大学）；China North Vehicle Research Institute（中国北方车辆研究所）
Road detection or traversability analysis has been a key technique for a mobile robot to traverse complex off-road scenes. The problem has been mainly formulated in early works as a binary classification one, e.g. associating pixels with road or non-road labels. Whereas understanding scenes with fine-grained labels are needed for off-road robots, as scenes are very diverse, and the various mechanical performance of off-road robots may lead to different definitions of safe regions to traverse. How to define and annotate fine-grained labels to achieve meaningful scene understanding for a robot t…
University of Lincoln（林肯大学）；University of Oxford（牛津大学）；Örebro University（厄勒布鲁大学）
This paper proposes a novel approach for global localisation of mobile robots in large-scale environments. Our method leverages learning-based localisation and filtering-based localisation, to localise the robot efficiently and precisely through seeding Monte Carlo Localisation (MCL) with a deep-learned distribution. In particular, a fast localisation system rapidly estimates the 6-DOF pose through a deep-probabilistic model (Gaussian Process Regression with a deep kernel), then a precise recursive estimator refines the estimated robot pose according to the geometric alignment. More importantl…
We propose a Doppler velocity-based cluster and velocity estimation algorithm based on the characteristics of FMCW LiDAR which achieves highly accurate, single-scan, and real-time motion state detection and velocity estimation. We prove the continuity of the Doppler velocity on the same object. Based on this principle, we achieve the distinction between moving objects and stationary background via region growing clustering algorithm. The obtained stationary background will be used to estimate the velocity of the FMCW LiDAR by the least-squares method. Then we estimate the velocity of the movin…
Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning
Korea Institute for Advanced Study（韩国高等研究院）；University of Seoul（首尔市立大学）；Seoul National University（首尔国立大学）；Hanyang University（汉阳大学）；Saige Research（赛奇研究院）
MINES ParisTech, PSL University, Centre for Robotics（巴黎高科矿业学院，巴黎文理研究大学，机器人中心）
KITTI-CARLA is a dataset built from the CARLA v0.9.10 simulator using a vehicle with sensors identical to the KITTI dataset. The vehicle thus has a Velodyne HDL64 LiDAR positioned in the middle of the roof and two color cameras similar to Point Grey Flea 2. The positions of the LiDAR and cameras are the same as the setup used in KITTI. The objective of this dataset is to test approaches of semantic segmentation LiDAR and/or images, odometry LiDAR and/or image in synthetic data and to compare with the results obtained on real data like KITTI. This dataset thus makes it possible to improve trans…
Deep reinforcement learning (RL), where the agent learns from mistakes, has been successfully applied to a variety of tasks. With the aim of learning collision-free policies for unmanned vehicles, deep RL has been used for training with various types of data, such as colored images, depth images, and LiDAR point clouds, without the use of classic map--localize--plan approaches. However, existing methods are limited by their reliance on cameras and LiDAR devices, which have degraded sensing under adverse environmental conditions (e.g., smoky environments). In response, we propose the use of sin…
MCTR: Midpoint Corrected Triangulation for Autonomous Racing via Digital Twin Simulation in CARLA
Department of Control Science and Engineering, Zhejiang University, Hangzhou, China（浙江大学控制科学与工程学院）；Ningbo Innovation Center, Zhejiang University, Ningbo, China（浙江大学宁波创新中心）；Center for Project-Based Learning, D-ITET, ETH Zurich（苏黎世联邦理工学院项目导向学习中心）；Technische Universität Wien, Vienna, Austria（维也纳技术大学）
本文提出 MCTR 算法，通过曲率校正移动平均改进轨迹平滑性，并在 CARLA 数字孪生中验证 3D LiDAR 感知下的自主赛车性能。
In this paper, a complete framework for Autonomous Self Driving is implemented. LIDAR, Camera and IMU sensors are used together. The entire data communication is managed using Robot Operating System which provides a robust platform for implementation of Robotics Projects. Jetson Nano is used to provide powerful on-board processing capabilities. Sensor fusion is performed on the data received from the different sensors to improve the accuracy of the decision making and inferences that we derive from the data. This data is then used to create a localized map of the environment. In this step, the…
To achieve fully autonomous navigation, vehicles need to compute an accurate model of their direct surrounding. In this paper, a 3D surface reconstruction algorithm from heterogeneous density 3D data is presented. The proposed method is based on a TSDF voxel-based representation, where an adaptive neighborhood kernel sourced on a Gaussian confidence evaluation is introduced. This enables to keep a good trade-off between the density of the reconstructed mesh and its accuracy. Experimental evaluations carried on both synthetic (CARLA) and real (KITTI) 3D data show a good performance compared to …
Massachusetts Institute of Technology（麻省理工学院）；Stevens Institute of Technology（史蒂文斯理工学院）
We propose a methodology for lidar super-resolution with ground vehicles driving on roadways, which relies completely on a driving simulator to enhance, via deep learning, the apparent resolution of a physical lidar. To increase the resolution of the point cloud captured by a sparse 3D lidar, we convert this problem from 3D Euclidean space into an image super-resolution problem in 2D image space, which is solved using a deep convolutional neural network. By projecting a point cloud onto a range image, we are able to efficiently enhance the resolution of such an image using a deep neural networ…
