Sparse-to-Continuous: Enhancing Monocular Depth Estimation using Occupancy Maps
University of São Paulo（圣保罗大学）；Toyota Research Institute（丰田研究院）
Nícolas Rosa, Vitor Guizilini, Valdir Grassi
备注：
Accepted. (c) 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
This paper addresses the problem of single image depth estimation (SIDE), focusing on improving the quality of deep neural network predictions. In a supervised learning scenario, the quality of predictions is intrinsically related to the training labels, which guide the optimization process. For indoor scenes, structured-light-based depth sensors (e.g. Kinect) are able to provide dense, albeit short-range, depth maps. On the other hand, for outdoor scenes, LiDARs are considered the standard sensor, which comparatively provides much sparser measurements, especially in areas further away. Rather than modifying the neural network architecture to deal with sparse depth maps, this article introduces a novel densification method for depth maps, using the Hilbert Maps framework. A continuous occupancy map is produced based on 3D points from LiDAR scans, and the resulting reconstructed surface is projected into a 2D depth map with arbitrary resolution. Experiments conducted with various subsets of the KITTI dataset show a significant improvement produced by the proposed Sparse-to-Continuous technique, without the introduction of extra information into the training stage.
Univrses AB（Univrses AB公司）；University of Bologna（博洛尼亚大学）
Estimating depth from a single image represents an attractive alternative to more traditional approaches leveraging multiple cameras. In this field, deep learning yielded outstanding results at the cost of needing large amounts of data labeled with precise depth measurements for training. An issue softened by self-supervised approaches leveraging monocular sequences or stereo pairs in place of expensive ground truth depth annotations. This paper enables to further improve monocular depth estimation by integrating into existing self-supervised networks a geometrical prior. Specifically, we prop…
We propose Ray-ONet to reconstruct detailed 3D models from monocular images efficiently. By predicting a series of occupancy probabilities along a ray that is back-projected from a pixel in the camera coordinate, our method Ray-ONet improves the reconstruction accuracy in comparison with Occupancy Networks (ONet), while reducing the network inference complexity to O($N^2$). As a result, Ray-ONet achieves state-of-the-art performance on the ShapeNet benchmark with more than 20$\times$ speed-up at $128^3$ resolution and maintains a similar memory footprint during inference.
EPRecon: An Efficient Framework for Real-Time Panoptic 3D Reconstruction from Monocular Video
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Monocular Depth Guided Occlusion-Aware Disparity Refinement via Semi-supervised Learning in Laparoscopic Images
School of Life Science and Technology, Harbin Institute of Technology（生命科学与技术学院，哈尔滨工业大学）；State Key Laboratory of Robotics and System, Harbin Institute of Technology（机器人与系统国家重点实验室，哈尔滨工业大学）
University of Oulu（奥卢大学）；Czech Technical University in Prague（布拉格捷克理工大学）；Tampere University（坦佩雷大学）
In this paper, we propose enhancing monocular depth estimation by adding 3D points as depth guidance. Unlike existing depth completion methods, our approach performs well on extremely sparse and unevenly distributed point clouds, which makes it agnostic to the source of the 3D points. We achieve this by introducing a novel multi-scale 3D point fusion network that is both lightweight and efficient. We demonstrate its versatility on two different depth estimation problems where the 3D points have been acquired with conventional structure-from-motion and LiDAR. In both cases, our network performs…
MonoOcc: Digging into Monocular Semantic Occupancy Prediction
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, the University of Hong Kong（香港大学计算机科学系）
Monocular Occupancy Prediction for Scalable Indoor Scenes
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；Centre for Artificial Intelligence and Robotics (HKISI_CAS)（人工智能与机器人中心）
Enhancing Monocular Height Estimation via Sparse LiDAR-Guided Correction
Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；RIKEN Center for Advanced Intelligence Project (AIP), RIKEN（日本理化学研究所先进智能项目中心（AIP））
Learning Explicit Continuous Motion Representation for Dynamic Gaussian Splatting from Monocular Videos
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（教育部机器智能与先进计算重点实验室）
Technical University of Munich（慕尼黑工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We introduce TransformerFusion, a transformer-based 3D scene reconstruction approach. From an input monocular RGB video, the video frames are processed by a transformer network that fuses the observations into a volumetric feature grid representing the scene; this feature grid is then decoded into an implicit 3D scene representation. Key to our approach is the transformer architecture that enables the network to learn to attend to the most relevant image frames for each 3D location in the scene, supervised only by the scene reconstruction task. Features are fused in a coarse-to-fine fashion, s…
DGNS: Deformable Gaussian Splatting and Dynamic Neural Surface for Monocular Dynamic 3D Reconstruction
Industrial Research Organisation, Canberra, ACT Australia；College of Engineering Computing \& Cybernetics, The Australian National University, Canberra, ACT Australia；Faculty of Engineering, The University of Hong Kong, Hong Kong SAR China；Industrial Research Organisation；College of Engineering Computing \& Cybernetics, The Australian National University；Faculty of Engineering, The University of Hong Kong
In this work, we research and evaluate end-to-end learning of monocular semantic-metric occupancy grid mapping from weak binocular ground truth. The network learns to predict four classes, as well as a camera to bird's eye view mapping. At the core, it utilizes a variational encoder-decoder network that encodes the front-view visual information of the driving scene and subsequently decodes it into a 2-D top-view Cartesian coordinate system. The evaluations on Cityscapes show that the end-to-end learning of semantic-metric occupancy grids outperforms the deterministic mapping approach with flat…
Monocular 3D scene understanding tasks, such as object size estimation, heading angle estimation and 3D localization, is challenging. Successful modern day methods for 3D scene understanding require the use of a 3D sensor. On the other hand, single image based methods have significantly worse performance. In this work, we aim at bridging the performance gap between 3D sensing and 2D sensing for 3D object detection by enhancing LiDAR-based algorithms to work with single image input. Specifically, we perform monocular depth estimation and lift the input image to a point cloud representation, whi…
MonoPartNeRF:Human Reconstruction from Monocular Video via Part-Based Neural Radiance Fields
The School of Computer Science and Information Security, Guilin University of Electronic Technology（桂林电子科技大学计算机科学与信息安全学院）；Metaverse Application Engineering Center（元宇宙应用工程中心）
Rethinking Transparent Object Grasping: Depth Completion with Monocular Depth Estimation and Instance Mask
State Key Laboratory of Robotics and System at Harbin Institute of Technology（哈尔滨工业大学机器人系统国家重点实验室）；Lanzhou University of Technology（兰州理工大学）；Department of Computer Science（计算机科学系）；University of Liverpool（利物浦大学）
Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能 MOE 实验室，上海交通大学人工智能学院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China（宁波数字孪生研究所，东技术研究所，宁波，中国）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Ningbo, China（宁波空间智能与数字衍生关键实验室，宁波，中国）；University of Science and Technology of China（中国科学技术大学）；CASIA；Tsinghua University（清华大学）
Autonomous driving requires efficient reasoning about the location and appearance of the different agents in the scene, which aids in downstream tasks such as object detection, object tracking, and path planning. The past few years have witnessed a surge in approaches that combine the different taskbased modules of the classic self-driving stack into an End-toEnd(E2E) trainable learning system. These approaches replace perception, prediction, and sensor fusion modules with a single contiguous module with shared latent space embedding, from which one extracts a human-interpretable representatio…
A Simple Framework for 3D Occupancy Estimation in Autonomous Driving
The University of Tokyo（东京大学）；RIKEN（理化学研究所）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；South China University of Technology（华南理工大学）
We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and g…
