Diffusion Models (DMs) have achieved State-Of-The-Art (SOTA) results in the Lidar point cloud generation task, benefiting from their stable training and iterative refinement during sampling. However, DMs often fail to realistically model Lidar raydrop noise due to their inherent denoising process. To retain the strength of iterative sampling while enhancing the generation of raydrop noise, we introduce LidarGRIT, a generative model that uses auto-regressive transformers to iteratively sample the range images in the latent space rather than image space. Furthermore, LidarGRIT utilises VQ-VAE to separately decode range images and raydrop masks. Our results show that LidarGRIT achieves superior performance compared to SOTA models on KITTI-360 and KITTI odometry datasets. Code available at:https://github.com/hamedhaghighi/LidarGRIT.
Microsoft Research（微软研究院）；Microsoft Mixed Reality & AI Lab（微软混合现实与人工智能实验室）；École polytechnique fédérale de Lausanne（洛桑联邦理工学院）
Simulating realistic sensors is a challenging part in data generation for autonomous systems, often involving carefully handcrafted sensor design, scene properties, and physics modeling. To alleviate this, we introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We propose a model that learns a mapping between RGB images and corresponding LiDAR features such as raydrop or per-point intensities directly from real datasets. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces or high intensity returns on reflective mat…
In the autonomous driving domain, data collection and annotation from real vehicles are expensive and sometimes unsafe. Simulators are often used for data augmentation, which requires realistic sensor models that are hard to formulate and model in closed forms. Instead, sensors models can be learned from real data. The main challenge is the absence of paired data set, which makes traditional supervised learning techniques not suitable. In this work, we formulate the problem as image translation from unpaired data and employ CycleGANs to solve the sensor modeling problem for LiDAR, to produce r…
Analyzing Infrastructure LiDAR Placement with Realistic LiDAR Simulation Library
Shanghai AI Laboratory（上海人工智能实验室）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We present LiDARGen, a novel, effective, and controllable generative model that produces realistic LiDAR point cloud sensory readings. Our method leverages the powerful score-matching energy-based model and formulates the point cloud generation process as a stochastic denoising process in the equirectangular view. This model allows us to sample diverse and high-quality point cloud samples with guaranteed physical feasibility and controllability. We validate the effectiveness of our method on the challenging KITTI-360 and NuScenes datasets. The quantitative and qualitative results show that our…
Technical University of Munich（慕尼黑工业大学）；Institute for Automotive Engineering, Munich Institute of Robotics and Machine Intelligence, School of Engineering and Design（汽车工程研究所，慕尼黑机器人与机器智能研究所，工程与设计学院）
Mercedes-Benz AG（梅赛德斯-奔驰股份公司）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Robert-Bosch GmbH（罗伯特·博世有限公司）；Research Center for Information Technology（信息技术研究中心）
A considerable amount of research is concerned with the generation of realistic sensor data. LiDAR point clouds are generated by complex simulations or learned generative models. The generated data is usually exploited to enable or improve downstream perception algorithms. Two major questions arise from these procedures: First, how to evaluate the realism of the generated data? Second, does more realistic data also lead to better perception performance? This paper addresses both questions and presents a novel metric to quantify the realism of LiDAR point clouds. Relevant features are learned f…
LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptation
China Ministry of Education (MOE) Key Laboratory of Artificial Intelligence（中国教育部人工智能重点实验室）；Artificial Intelligence Institute, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Huawei Noah’s Ark Lab（华为诺亚实验室）
The University of Adelaide（阿德莱德大学）；ETH Zurich（苏黎世联邦理工学院）
Point cloud registration is a fundamental problem in 3D scanning. In this paper, we address the frequent special case of registering terrestrial LiDAR scans (or, more generally, levelled point clouds). Many current solutions still rely on the Iterative Closest Point (ICP) method or other heuristic procedures, which require good initializations to succeed and/or provide no guarantees of success. On the other hand, exact or optimal registration algorithms can compute the best possible solution without requiring initializations; however, they are currently too slow to be practical in realistic ap…
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）；CWB Institute of Autonomous Driving（CWB自动驾驶研究院）
The combination of LiDARs and cameras enables a mobile robot to perceive environments with multi-modal data, becoming a key factor in achieving robust perception. Traditional frame cameras are sensitive to changing illumination conditions, motivating us to introduce novel event cameras to make LiDAR-camera fusion more complete and robust. However, to jointly exploit these sensors, the challenging extrinsic calibration problem should be addressed. This paper proposes an automatic checkerboard-based approach to calibrate extrinsics between a LiDAR and a frame/event camera, where four contributio…
Following improvements in deep neural networks, state-of-the-art networks have been proposed for human recognition using point clouds captured by LiDAR. However, the performance of these networks strongly depends on the training data. An issue with collecting training data is labeling. Labeling by humans is necessary to obtain the ground truth label; however, labeling requires huge costs. Therefore, we propose an automatic labeled data generation pipeline, for which we can change any parameters or data generation environments. Our approach uses a human model named Dhaiba and a background of Mi…
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；College of Electronic Science and Technology, National University of Defense Technology（国防科技大学电子科学学院）
LiDAR sensors can provide dependable 3D spatial information at a low frequency (around 10Hz) and have been widely applied in the field of autonomous driving and UAV. However, the camera with a higher frequency (around 20Hz) has to be decreased so as to match with LiDAR in a multi-sensor system. In this paper, we propose a novel Pseudo-LiDAR interpolation network (PLIN) to increase the frequency of LiDAR sensors. PLIN can generate temporally and spatially high-quality point cloud sequences to match the high frequency of cameras. To achieve this goal, we design a coarse interpolation stage guide…
KAIST (Korea Advanced Institute of Science and Technology)（韩国科学技术院）
In this study, a deep-learning-based multi-stage network architecture called Multi-Stage Depth Prediction Network (MSDPN) is proposed to predict a dense depth map using a 2D LiDAR and a monocular camera. Our proposed network consists of a multi-stage encoder-decoder architecture and Cross Stage Feature Aggregation (CSFA). The proposed multi-stage encoder-decoder architecture alleviates the partial observation problem caused by the characteristics of a 2D LiDAR, and CSFA prevents the multi-stage network from diluting the features and allows the network to learn the inter-spatial relationship be…
In this paper, we address semantic segmentation of road-objects from 3D LiDAR point clouds. In particular, we wish to detect and categorize instances of interest, such as cars, pedestrians and cyclists. We formulate this problem as a point- wise classification problem, and propose an end-to-end pipeline called SqueezeSeg based on convolutional neural networks (CNN): the CNN takes a transformed LiDAR point cloud as input and directly outputs a point-wise label map, which is then refined by a conditional random field (CRF) implemented as a recurrent layer. Instance-level labels are then obtained…
Tokyo Institute of Technology（东京工业大学）；National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）
In this paper, we propose an automatic labeled sequential data generation pipeline for human segmentation and velocity estimation with point clouds. Considering the impact of deep neural networks, state-of-the-art network architectures have been proposed for human recognition using point clouds captured by Light Detection and Ranging (LiDAR). However, one disadvantage is that legacy datasets may only cover the image domain without providing important label information and this limitation has disturbed the progress of research to date. Therefore, we develop an automatic labeled sequential data …
A Graph Attention Network-Based Framework for Reconstructing Missing LiDAR Beams
Computer Information Technology Department, SUNY Morrisville（SUNY Morrisville 计算机信息科技系）；Electrical and Computer Engineering Department, Portland State University（波特兰州立大学电气与计算机工程系）；Electrical and Computer Engineering Department, Lawrence Technological University（劳伦斯技术大学电气与计算机工程系）
This work introduces an evaluation benchmark for depth estimation and completion using high-resolution depth measurements with angular resolution of up to 25" (arcsecond), akin to a 50 megapixel camera with per-pixel depth available. Existing datasets, such as the KITTI benchmark, provide only sparse reference measurements with an order of magnitude lower angular resolution - these sparse measurements are treated as ground truth by existing depth estimation methods. We propose an evaluation methodology in four characteristic automotive scenarios recorded in varying weather conditions (day, nig…
A Unified Generative Framework for Realistic Lidar Simulation in Autonomous Driving Systems
WMG, University of Warwick（华威大学WMG学院）；School of Electronics, Electrical Engineering and Computer Science at Queen’s University of Belfast（贝尔法斯特女王大学电子、电气工程与计算机科学学院）
Uber Advanced Technologies Group（优步先进技术集团）；University of Toronto（多伦多大学）；Massachusetts Institute of Techonology（麻省理工学院）
We tackle the problem of producing realistic simulations of LiDAR point clouds, the sensor of preference for most self-driving vehicles. We argue that, by leveraging real data, we can simulate the complex world more realistically compared to employing virtual worlds built from CAD/procedural models. Towards this goal, we first build a large catalog of 3D static maps and 3D dynamic objects by driving around several cities with our self-driving fleet. We can then generate scenarios by selecting a scene from our catalog and "virtually" placing the self-driving vehicle (SDV) and a set of dynamic o…
Shape completion, the problem of estimating the complete geometry of objects from partial observations, lies at the core of many vision and robotics applications. In this work, we propose Point Completion Network (PCN), a novel learning-based approach for shape completion. Unlike existing shape completion methods, PCN directly operates on raw point clouds without any structural assumption (e.g. symmetry) or annotation (e.g. semantic class) about the underlying shape. It features a decoder design that enables the generation of fine-grained completions while maintaining a small number of paramet…
We examined the feasibility of generative adversarial networks (GANs) to generate photo-realistic images from LiDAR point clouds. For this purpose, we created a dataset of point cloud image pairs and trained the GAN to predict photorealistic images from LiDAR point clouds containing reflectance and distance information. Our models learned how to predict realistically looking images from just point cloud data, even images with black cars. Black cars are difficult to detect directly from point clouds because of their low level of reflectivity. This approach might be used in the future to perform…
Google Research（谷歌研究院）；University of Toronto（多伦多大学）
The goal of this work is to perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments (e.g., Street View). Given a sequence of posed RGB images and lidar sweeps acquired by cameras and scanners moving through an outdoor scene, we produce a model from which 3D surfaces can be extracted and novel RGB images can be synthesized. Our approach extends Neural Radiance Fields, which has been demonstrated to synthesize realistic novel images for small scenes in controlled settings, with new methods for l…
EvidenceMoE: A Physics-Guided Mixture-of-Experts with Evidential Critics for Advancing Fluorescence Light Detection and Ranging in Scattering Media
Center for Modeling, Simulation, & Imaging in Medicine（建模、仿真与成像医学中心）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；University at Albany（阿尔巴尼大学）；IBM T.J. Watson Research Center（IBM 沃森研究中心）
MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）；Academy of Military Science（军事科学院）；Bosch innovation software development (Wuxi) Co., Ltd.（博世创新软件开发（无锡）有限公司）；College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机学院）；Shopee Pte. Ltd.（Shopee私人有限公司）
SN-LiDAR: Semantic Neural Fields for Novel Space-time View LiDAR Synthesis
Shanghai Jiao Tong University（上海交通大学）；Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院）；Renji Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属仁济医院）
Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology（东部技术研究院）；School of Electronic Information and Electrical Engineering（电子信息与电气工程学院）；Li Auto（力汽车）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生实验室）；Ningbo Institute of Digital Twin（宁波数字孪生研究院）
Real-Aug: Realistic Scene Synthesis for LiDAR Augmentation in 3D Object Detection
Inspur Electronic Information Industry Co.,Ltd.（浪潮电子信息产业股份有限公司）；Centre for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港创新研究院人工智能与机器人中心）；Institute of Automation, CAS（中国科学院自动化研究所）
T2LDM++: A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation
School of Computer Science and Engineering（计算机科学与工程学院）；Nanjing University of Science and Technology（南京理工大学）；Faculty of Data Science（数据科学学院）；City University of Macau（澳门城市大学）；Beijing Institute of Technology（北京理工大学）；Fondazione Bruno Kessler（布鲁诺·凯瑟尔基金会）；School of Public Health（公共卫生学院）；Shanghai Jiao Tong University（上海交通大学）；School of Computing（计算机学院）；National University of Singapore（新加坡国立大学）
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）
Fusing the camera and LiDAR information has become a de-facto standard for 3D object detection tasks. Current methods rely on point clouds from the LiDAR sensor as queries to leverage the feature from the image space. However, people discovered that this underlying assumption makes the current fusion framework infeasible to produce any prediction when there is a LiDAR malfunction, regardless of minor or major. This fundamentally limits the deployment capability to realistic autonomous driving scenarios. In contrast, we propose a surprisingly simple yet novel fusion framework, dubbed BEVFusion,…
Supervised deep learning often suffers from the lack of sufficient training data. Specifically in the context of monocular depth map prediction, it is barely possible to determine dense ground truth depth images in realistic dynamic outdoor environments. When using LiDAR sensors, for instance, noise is present in the distance measurements, the calibration between sensors cannot be perfect, and the measurements are typically much sparser than the camera images. In this paper, we propose a novel approach to depth map prediction from monocular images that learns in a semi-supervised way. While we…
