In this paper, we propose an end-to-end deep learning network named 3dDepthNet, which produces an accurate dense depth image from a single pair of sparse LiDAR depth and color image for robotics and autonomous driving tasks. Based on the dimensional nature of depth images, our network offers a novel 3D-to-2D coarse-to-fine dual densification design that is both accurate and lightweight. Depth densification is first performed in 3D space via point cloud completion, followed by a specially designed encoder-decoder structure that utilizes the projected dense depth from 3D completion and the original RGB-D images to perform 2D image completion. Experiments on the KITTI dataset show our network achieves state-of-art accuracy while being more efficient. Ablation and generalization tests prove that each module in our network has positive influences on the final results, and furthermore, our network is resilient to even sparser depth.
Tsinghua University（清华大学）；Huawei Technologies Canada Co Ltd（华为技术加拿大有限公司）；OPPO Research Institute（OPPO研究院）
This paper presents a view-guided solution for the task of point cloud completion. Unlike most existing methods directly inferring the missing points using shape priors, we address this task by introducing ViPC (view-guided point cloud completion) that takes the missing crucial global structure information from an extra single-view image. By leveraging a framework that sequentially performs effective cross-modality and cross-level fusions, our method achieves significantly superior results over typical existing solutions on a new large-scale dataset we collect for the view-guided point cloud c…
MAGE: View-guided Point Cloud Completion with Efficient Modality Alignment and Adaptive Geometry Enhancement
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS研究室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；GIPSA-lab, Univ. Grenoble Alpes, CNRS, Grenoble INP（格勒诺布尔大学GIPSA实验室，法国国家科学研究中心，格勒诺布尔国立综合理工学院）
Fine-grained Text and Image Guided Point Cloud Completion with CLIP Model
Dalian Maritime University（大连海事大学）；Zhejiang Sci-Tech University（浙江理工大学）；Beijing University of Technology（北京工业大学）；Dalian University of Technology（大连理工大学）
Memory-Guided Point Cloud Completion for Dental Reconstruction
College of Information and Intelligent Science, Donghua University, Shanghai, China（信息与智能科学学院，东华大学，上海，中国）；Institute of Artificial Intelligence, Shanghai University, Shanghai, China（人工智能研究院，上海大学，上海，中国）
In this paper we explore the recent topic of point cloud completion, guided by an auxiliary image. We show how it is possible to effectively combine the information from the two modalities in a localized latent space, thus avoiding the need for complex point cloud reconstruction methods from single views used by the state-of-the-art. We also investigate a novel weakly-supervised setting where the auxiliary image provides a supervisory signal to the training process by using a differentiable renderer on the completed point cloud to measure fidelity in the image space. Experiments show significa…
The Chinese University of Hong Kong(Shenzhen)（香港中文大学（深圳））；Dalian University of Technology（大连理工大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；Texas A&M University（得克萨斯农工大学）
We present a deep reinforcement learning method of progressive view inpainting for 3D point scene completion under volume guidance, achieving high-quality scene reconstruction from only a single depth image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D depth map inpainting, and multi-view selection for completion. Given a single depth image, our method first goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide to the next view inpainting step, which attempts to make up the missing informati…
Dalian University of Technology（大连理工大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong (ShenZhen)（香港中文大学（深圳））；Princeton University（普林斯顿大学）
We present a deep reinforcement learning method of progressive view inpainting for colored semantic point cloud scene completion under volume guidance, achieving high-quality scene reconstruction from only a single RGB-D image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D RGB-D and segmentation image inpainting, and multi-view selection for completion. Given a single RGB-D image, our method first predicts its semantic segmentation map and goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide…
Hong Kong University of Science and Technology（香港科技大学）；DJI Co.（大疆公司）
For a long time, the point cloud completion task has been regarded as a pure generation task. After obtaining the global shape code through the encoder, a complete point cloud is generated using the shape priorly learnt by the networks. However, such models are undesirably biased towards prior average objects and inherently limited to fit geometry details. In this paper, we propose a Graph-Guided Deformation Network, which respectively regards the input data and intermediate generation as controlling and supporting points, and models the optimization guided by a graph convolutional network(GCN…
Toward Patient-specific Partial Point Cloud to Surface Completion for Pre- to Intra-operative Registration in Image-guided Liver Interventions
Center for Imaging Science, Rochester Institute of Technology, Rochester, NY 14623, USA（成像科学中心，罗切斯特理工学院，罗切斯特，NY 14623, USA）；Biomedical Engineering, Rochester Institute of Technology, Rochester, NY 14623, USA（生物医学工程，罗切斯特理工学院，罗切斯特，NY 14623, USA）
Epson Research（爱普生研究院）；University of Toronto（多伦多大学）
Depth completion aims at inferring a dense depth image from sparse depth measurement since glossy, transparent or distant surface cannot be scanned properly by the sensor. Most of existing methods directly interpolate the missing depth measurements based on pixel-wise image content and the corresponding neighboring depth values. Consequently, this leads to blurred boundaries or inaccurate structure of object. To address these problems, we propose a novel self-guided instance-aware network (SG-IANet) that: (1) utilize self-guided mechanism to extract instance-level features that is needed for d…
Position-aware Guided Point Cloud Completion with CLIP Model
North China University of Technology（北方工业大学）；Peng Cheng Laboratory（鹏城实验室）；University of Copenhagen（哥本哈根大学）；University of Washington（华盛顿大学）；Skywork AI（天工智能）；Tsinghua University（清华大学）
Depth completion aims to generate a dense depth map from the sparse depth map and aligned RGB image. However, current depth completion methods use extremely expensive 64-line LiDAR(about $100,000) to obtain sparse depth maps, which will limit their application scenarios. Compared with the 64-line LiDAR, the single-line LiDAR is much less expensive and much more robust. Therefore, we propose a method to tackle the problem of single-line depth completion, in which we aim to generate a dense depth map from the single-line LiDAR info and the aligned RGB image. A single-line depth completion datase…
A Strong View-Free Baseline Approach for Single-View Image Guided Point Cloud Completion
Department of Robotics Engineering, Worcester Polytechnic Institute（机器人工程系，沃斯特理工学院）；Dell Technologies（戴尔技术）；Graduate School of Information Sciences, Tohoku University（信息科学研究生院，东北大学）；Department of Computer Science, Worcester Polytechnic Institute（计算机科学系，沃斯特理工学院）；Harvard Kenneth C. Griffin Graduate School of Arts and Sciences（哈佛肯尼斯·C·格里芬艺术与科学研究生院）；Department of Biomedical Engineering, Worcester Polytechnic Institute（生物医学工程系，沃斯特理工学院）；Department of Electrical & Computer Engineering, Worcester Polytechnic Institute（电气与计算机工程系，沃斯特理工学院）
University of Tokyo（东京大学）；Nippon Telegraph and Telephone Corporation（日本电报电话公司）
We propose an unsupervised real-time dense depth completion from a sparse depth map guided by a single image. Our method generates a smooth depth map while preserving discontinuity between different objects. Our key idea is a Binary Anisotropic Diffusion Tensor (B-ADT) which can completely eliminate smoothness constraint at intended positions and directions by applying it to variational regularization. We also propose an Image-guided Nearest Neighbor Search (IGNNS) to derive a piecewise constant depth map which is used for B-ADT derivation and in the data term of the variational energy. Our ex…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
The paper proposes an image-guided depth completion method to estimate accurate dense depth maps with fast computation time. The proposed network has two-stage structure. The first stage predicts a first depth map. Then, the second stage further refines the first depth map using the confidence maps. The second stage consists of two layers, each of which focuses on different regions and generates a refined depth map and a confidence map. The final depth map is obtained by combining two depth maps from the second stage using the corresponding confidence maps. Compared with the top-ranked models …
We present a new flow-based video completion algorithm. Previous flow completion methods are often unable to retain the sharpness of motion boundaries. Our method first extracts and completes motion edges, and then uses them to guide piecewise-smooth flow completion with sharp edges. Existing methods propagate colors among local flow connections between adjacent frames. However, not all missing regions in a video can be reached in this way because the motion boundaries form impenetrable barriers. Our method alleviates this problem by introducing non-local flow connections to temporally distant…
Northwestern Polytechnical University（西北工业大学）；Australian National University（澳大利亚国立大学）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织 Data61）
We aim at predicting a complete and high-resolution depth map from incomplete, sparse and noisy depth measurements. Existing methods handle this problem either by exploiting various regularizations on the depth maps directly or resorting to learning based methods. When the corresponding color images are available, the correlation between the depth maps and the color images are used to improve the completion performance, assuming the color images are clean and sharp. However, in real world dynamic scenes, color images are often blurry due to the camera motion and the moving objects in the scene…
Nanjing University of Science and Technology（南京理工大学）
Depth completion deals with the problem of recovering dense depth maps from sparse ones, where color images are often used to facilitate this task. Recent approaches mainly focus on image guided learning frameworks to predict dense depth. However, blurry guidance in the image and unclear structure in the depth still impede the performance of the image guided frameworks. To tackle these problems, we explore a repetitive design in our image guided network to gradually and sufficiently recover depth values. Specifically, the repetition is embodied in both the image guidance branch and depth gener…
SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completion
Department of Computer Science, University of Kaiserslautern, 67663 Kaiserslautern, Germany（凯斯莱特大学计算机科学系）；Mindgrage, University of Kaiserslautern, 67663 Kaiserslautern, Germany（Mindgrage凯斯莱特大学）；Department of Computer Science, Luleå University of Technology, 971 87 Luleå, Sweden（卢勒奥技术大学计算机科学系）
University of Rochester（罗切斯特大学）；University of Chinese Academy of Sciences（中国科学院大学）；University of North Texas（北得克萨斯大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
Depth Completion can produce a dense depth map from a sparse input and provide a more complete 3D description of the environment. Despite great progress made in depth completion, the sparsity of the input and low density of the ground truth still make this problem challenging. In this work, we propose DenseLiDAR, a novel real-time pseudo-depth guided depth completion neural network. We exploit dense pseudo-depth map obtained from simple morphological operations to guide the network in three aspects: (1) Constructing a residual structure for the output; (2) Rectifying the sparse input data; (3)…
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；Hisilicon, Huawei（华为海思）
Image guided depth completion is the task of generating a dense depth map from a sparse depth map and a high quality image. In this task, how to fuse the color and depth modalities plays an important role in achieving good performance. This paper proposes a two-branch backbone that consists of a color-dominant branch and a depth-dominant branch to exploit and fuse two modalities thoroughly. More specifically, one branch inputs a color image and a sparse depth map to predict a dense depth map. The other branch takes as inputs the sparse depth map and the previously predicted depth map, and outp…
In this paper we propose a convolutional neural network that is designed to upsample a series of sparse range measurements based on the contextual cues gleaned from a high resolution intensity image. Our approach draws inspiration from related work on super-resolution and in-painting. We propose a novel architecture that seeks to pull contextual cues separately from the intensity image and the depth features and then fuse them later in the network. We argue that this approach effectively exploits the relationship between the two modalities and produces accurate results while respecting salient…
Simon Fraser University（西蒙弗雷泽大学）；National University of Defense Technology（国防科技大学）；Tianjin University（天津大学）；State Administration of Cultural Heritage（国家文物局）
Dense depth perception is critical for autonomous driving and other robotics applications. However, modern LiDAR sensors only provide sparse depth measurement. It is thus necessary to complete the sparse LiDAR data, where a synchronized guidance RGB image is often used to facilitate this completion. Many neural networks have been designed for this task. However, they often na\"ıvely fuse the LiDAR data and RGB image information by performing feature concatenation or element-wise addition. Inspired by the guided image filtering, we design a novel guided network to predict kernel weights from th…
DFKI - German Research Center for Artificial Intelligence（德国人工智能研究中心）；BMW Group（宝马集团）
Interpolation of sparse pixel information towards a dense target resolution finds its application across multiple disciplines in computer vision. State-of-the-art interpolation of motion fields applies model-based interpolation that makes use of edge information extracted from the target image. For depth completion, data-driven learning approaches are widespread. Our work is inspired by latest trends in depth completion that tackle the problem of dense guidance for sparse information. We extend these ideas and create a generic cross-domain architecture that can be applied for a multitude of in…
Sparse Depth-Guided Attention for Accurate Depth Completion: A Stereo-Assisted Monitored Distillation Approach
Huawei Cloud Computing Technology Co., Ltd.（华为云计算技术有限公司）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人学会）；Institute of Robotics and Intelligent Manufacturing, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）机器人与智能制造研究院）
Mask Guided Gated Convolution for Amodal Content Completion
Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University（奥布达大学应用信息学与应用数学博士学院）；John von Neumann Faculty of Informatics, Óbuda University（奥布达大学约翰·冯·诺依曼信息学院）
Acquiring complete and clean 3D shape and scene data is challenging due to geometric occlusion and insufficient views during 3D capturing. We present a simple yet effective deep learning approach for completing the input noisy and incomplete shapes or scenes. Our network is built upon the octree-based CNNs (O-CNN) with U-Net like structures, which enjoys high computational and memory efficiency and supports to construct a very deep network structure for 3D CNNs. A novel output-guided skip-connection is introduced to the network structure for better preserving the input geometry and learning ge…
CurriFlow: Curriculum-Guided Depth Fusion with Optical Flow-Based Temporal Alignment for 3D Semantic Scene Completion
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Ministry of Education（教育部）；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东省计算机科学中心（济南国家超算中心））；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing（山东省计算电力互联网与服务计算重点实验室）；Shandong Fundamental Research Center for Computer Science（山东省计算机科学基础研究中心）
UC Santa Barbara（加州大学圣巴巴拉分校）；Meta；NEC Laboratories America（NEC美国实验室）
Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence. Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion. Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability. Inspired by this, we introduce a novel task, text-guided video completion (TVC), which re…
A Concise but High-performing Network for Image Guided Depth Completion in Autonomous Driving
School of Mechanical Science and Engineering, Huazhong University of Science and Technology（华中科技大学机械科学与工程学院）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局前沿人工智能研究中心）；Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（香港理工大学电机工程学系）；School of Mechanical Engineering, Hubei University of Technology（湖北工业大学机械工程学院）
Xi’an University of Science and Technology（西安科技大学）
A novel color image enhancement method is proposed based on Retinex to enhance color images under non-uniform illumination or poor visibility conditions. Different from the conventional Retinex algorithms, the Weighted Guided Image Filter is used as a surround function instead of the Gaussian filter to estimate the background illumination, which can overcome the drawbacks of local blur and halo artifact that may appear by Gaussian filter. To avoid color distortion, the image is converted to the HSI color model, and only the intensity channel is enhanced. Then a linear color restoration algorit…
