Recent advances in neural radiance fields (NeRFs) achieve state-of-the-art novel view synthesis and facilitate dense estimation of scene properties. However, NeRFs often fail for large, unbounded scenes that are captured under very sparse views with the scene content concentrated far away from the camera, as is typical for field robotics applications. In particular, NeRF-style algorithms perform poorly: (1) when there are insufficient views with little pose diversity, (2) when scenes contain saturation and shadows, and (3) when finely sampling large unbounded scenes with fine structures becomes computationally intensive.
This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes that are observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for volumetric rendering in metric space.
Through extensive quantitative and qualitative experiments on scenes from the KITTI dataset, this paper demonstrates that the proposed method outperforms state-of-the-art NeRF models on both novel view synthesis and dense depth prediction tasks when trained on sparse input data.
LASTIG, Univ. Gustave Eiffel, IGN-ENSG（古斯塔夫·埃菲尔大学LASTIG实验室，法国国家地理与林业信息学院）；INRAE, UMR 1202 BIOGECO, Université de Bordeaux（法国国家农业食品与环境研究院，波尔多大学生物地球化学与生态学联合研究中心）；EA G&E Bordeaux INP, Université Bordeaux Montaigne（波尔多蒙田大学，波尔多国立高等电子、计算机、电信、数学与机械学院环境与地理空间工程研究团队）
We propose a new deep learning-based method for estimating the occupancy of vegetation strata from airborne 3D LiDAR point clouds. Our model predicts rasterized occupancy maps for three vegetation strata corresponding to lower, medium, and higher cover. Our weakly-supervised training scheme allows our network to only be supervised with vegetation occupancy values aggregated over cylindrical plots containing thousands of points. Such ground truth is easier to produce than pixel-wise or point-wise annotations. Our method outperforms handcrafted and deep learning baselines in terms of precision b…
Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology（东部技术研究院）；School of Electronic Information and Electrical Engineering（电子信息与电气工程学院）；Li Auto（力汽车）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生实验室）；Ningbo Institute of Digital Twin（宁波数字孪生研究院）
University of São Paulo（圣保罗大学）；Toyota Research Institute（丰田研究院）
This paper addresses the problem of single image depth estimation (SIDE), focusing on improving the quality of deep neural network predictions. In a supervised learning scenario, the quality of predictions is intrinsically related to the training labels, which guide the optimization process. For indoor scenes, structured-light-based depth sensors (e.g. Kinect) are able to provide dense, albeit short-range, depth maps. On the other hand, for outdoor scenes, LiDARs are considered the standard sensor, which comparatively provides much sparser measurements, especially in areas further away. Rather…
Although convolutional representation of multiscale sparse tensor demonstrated its superior efficiency to accurately model the occupancy probability for the compression of geometry component of dense object point clouds, its capacity for representing sparse LiDAR point cloud geometry (PCG) was largely limited. This is because 1) fixed receptive field of the convolution cannot characterize extremely and unevenly distributed sparse LiDAR points very well; and 2) pretrained convolutions with fixed weights are insufficient to dynamically capture information conditioned on the input. This work ther…
Telekooperation Lab, Technical University Darmstadt（达姆施塔特工业大学远程协作实验室）
In this work, we address the problem of unsupervised moving object segmentation (MOS) in 4D LiDAR data recorded from a stationary sensor, where no ground truth annotations are involved. Deep learning-based state-of-the-art methods for LiDAR MOS strongly depend on annotated ground truth data, which is expensive to obtain and scarce in existence. To close this gap in the stationary setting, we propose a novel 4D LiDAR representation based on multivariate time series that relaxes the problem of unsupervised MOS to a time series clustering problem. More specifically, we propose modeling the change…
Université Gustave Eiffel, IGN-ENSG, LASTIG（古斯塔夫·埃菲尔大学，法国国家地理与林业信息研究所-国家地理科学学院，地理信息与空间分析实验室）；INRAE, UMR 1202 BIOGECO, Université de Bordeaux（法国国家农业食品与环境研究院，BIOGECO联合研究单位，波尔多大学）；EA G&E Bordeaux INP, Université Bordeaux Montaigne（波尔多综合理工学院G&E研究组，波尔多蒙田大学）
We propose a new deep learning-based method for estimating the occupancy of vegetation strata from 3D point clouds captured from an aerial platform. Our model predicts rasterized occupancy maps for three vegetation strata: lower, medium, and higher strata. Our training scheme allows our network to only being supervized with values aggregated over cylindrical plots, which are easier to produce than pixel-wise or point-wise annotations. Our method outperforms handcrafted and deep learning baselines in terms of precision while simultaneously providing visual and interpretable predictions. We prov…
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；FZI Research Center for Information Technology（FZI信息技术研究中心）
We present a generic evidential grid mapping pipeline designed for imaging sensors such as LiDARs and cameras. Our grid-based evidential model contains semantic estimates for cell occupancy and ground separately. We specify the estimation steps for input data represented by point sets, but mainly focus on input data represented by images such as disparity maps or LiDAR range images. Instead of relying on an external ground segmentation only, we deduce occupancy evidence by analyzing the surface orientation around measurements. We conduct experiments and evaluate the presented method using LiDA…
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；The University of Hong Kong（香港大学）；Hong Kong University of Science and Technology（香港科技大学）；DJI Co.（大疆创新科技有限公司）
Modeling scene geometry using implicit neural representation has revealed its advantages in accuracy, flexibility, and low memory usage. Previous approaches have demonstrated impressive results using color or depth images but still have difficulty handling poor light conditions and large-scale scenes. Methods taking global point cloud as input require accurate registration and ground truth coordinate labels, which limits their application scenarios. In this paper, we propose a new method that uses sparse LiDAR point clouds and rough odometry to reconstruct fine-grained implicit occupancy field…
Co-Occ: Coupling Explicit Feature Fusion with Volume Rendering Regularization for Multi-Modal 3D Semantic Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；HKUST(GZ)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）
MS-Occ: Multi-Stage LiDAR-Camera Fusion for 3D Semantic Occupancy Prediction
School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology（光学电子与计算机工程学院，上海科学技术大学）；School of Automotive Studies, Tongji University（汽车学院，同济大学）；Momoni AI；College of Science and Engineering, James Cook University（科学与工程学院，詹姆斯库克大学）；School of Engineering, Swinburne University of Technology（工程学院，斯威本技术大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电气与电子工程学院，南洋理工大学）
UniScene: Unified Occupancy-centric Driving Scene Generation
Shanghai Jiao Tong University（上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Tsinghua University（清华大学）；MEGVII Technology（旷视科技）；Mach Drive（马赫驱动）；Fudan University（复旦大学）；University of Hong Kong（香港大学）
Lidar datasets are becoming more and more common. They are appreciated for their precise 3D nature, and have a wide range of applications, such as surface reconstruction, object detection, visualisation, etc. For all this applications, having additional semantic information per point has potential of increasing the quality and the efficiency of the application. In the last decade the use of Machine Learning and more specifically classification methods have proved to be successful to create this semantic information. In this paradigm, the goal is to classify points into a set of given classes (…
SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；School of Information Science and Technology（信息科学与技术学院）；Fudan University（复旦大学）；School of Artificial Intelligence（人工智能学院）；Shanghai Jiao Tong University（上海交通大学）
Université Gustave Eiffel（古斯塔夫·埃菲尔大学）；ENSG（法国国立地理科学学院）；IGN（法国国家地理和森林信息研究所）；INRAE（法国国家农业、食品与环境研究院）；Université de Bordeaux（波尔多大学）；Bordeaux INP（波尔多国立理工学院）；Université Bordeaux Montaigne（波尔多蒙田大学）
The analysis of the multi-layer structure of wild forests is an important challenge of automated large-scale forestry. While modern aerial LiDARs offer geometric information across all vegetation layers, most datasets and methods focus only on the segmentation and reconstruction of the top of canopy. We release WildForest3D, which consists of 29 study plots and over 2000 individual trees across 47 000m2 with dense 3D annotation, along with occupancy and height maps for 3 vegetation layers: ground vegetation, understory, and overstory. We propose a 3D deep network architecture predicting for th…
GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；School of Electronics, Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Transportation and Traffic Engineering, Central South University（中南大学交通运输工程学院）
Gau-Occ: Geometry-Completed Gaussians for Multi-Modal 3D Occupancy Prediction
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China（虚拟现实技术与系统国家重点实验室，北京航空航天大学，北京，中国）；School of Computer Science and Engineering, Beihang University, Beijing, China（计算机科学与工程学院，北京航空航天大学，北京，中国）；School of Artificial Intelligence, Beihang University, Beijing, China（人工智能学院，北京航空航天大学，北京，中国）
LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学 cyber 系统与控制研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Technical University of Munich（慕尼黑工业大学）
Detecting moving objects in dynamic scenes from sequences of lidar scans is an important task in object tracking, mapping, localization, and navigation. Many works focus on changes detection in previously observed scenes, while a very limited amount of literature addresses moving objects detection. The state-of-the-art method exploits Dempster-Shafer Theory to evaluate the occupancy of a lidar scan and to discriminate points belonging to the static scene from moving ones. In this paper we improve both speed and accuracy of this method by discretizing the occupancy representation, and by removi…
General Motors, Advanced Technical Center（通用汽车先进技术中心）
Occupancy grid mapping is an important component in road scene understanding for autonomous driving. It encapsulates information of the drivable area, road obstacles and enables safe autonomous driving. Radars are an emerging sensor in autonomous vehicle vision, becoming more widely used due to their long range sensing, low cost, and robustness to severe weather conditions. Despite recent advances in deep learning technology, occupancy grid mapping from radar data is still mostly done using classical filtering approaches.In this work, we propose learning the inverse sensor model used for occup…
Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene not only changes with the motion of actors in the environment, but also with the movement of the ego-vehicle itself. Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners. In this paper, we use geometric occupancy as a natural alternative to view-dependent representations suc…
TU Dortmund（多特蒙德工业大学）；HELLA Aglaia Mobile Vision GmbH（海拉阿格莱亚移动视觉有限公司）
Environment modeling utilizing sensor data fusion and object tracking is crucial for safe automated driving. In recent years, the classical occupancy grid map approach, which assumes a static environment, has been extended to dynamic occupancy grid maps, which maintain the possibility of a low-level data fusion while also estimating the position and velocity distribution of the dynamic local environment. This paper presents the further development of a previous approach. To the best of the author's knowledge, there is no publication about dynamic occupancy grid mapping with subsequent analysis…
Institute of Measurement, Control, and Microtechnology, Ulm University（乌尔姆大学测量、控制与微技术研究所）
We tackle the long-term prediction of scene evolution in a complex downtown scenario for automated driving based on Lidar grid fusion and recurrent neural networks (RNNs). A bird's eye view of the scene, including occupancy and velocity, is fed as a sequence to a RNN which is trained to predict future occupancy. The nature of prediction allows generation of multiple hours of training data without the need of manual labeling. Thus, the training strategy and loss function is designed for long sequences of real-world data (unbalanced, continuously changing situations, false labels, etc.). The dee…
