Understanding the flow in 3D space of sparsely sampled points between two consecutive time frames is the core stone of modern geometric-driven systems such as VR/AR, Robotics, and Autonomous driving. The lack of real, non-simulated, labeled data for this task emphasizes the importance of self- or un-supervised deep architectures. This work presents a new self-supervised training method and an architecture for the 3D scene flow estimation under occlusions. Here we show that smart multi-layer fusion between flow prediction and occlusion detection outperforms traditional architectures by a large margin for occluded and non-occluded scenarios. We report state-of-the-art results on Flyingthings3D and KITTI datasets for both the supervised and self-supervised training.
Self-supervised deep learning-based 3D scene understanding methods can overcome the difficulty of acquiring the densely labeled ground-truth and have made a lot of advances. However, occlusions and moving objects are still some of the major limitations. In this paper, we explore the learnable occlusion aware optical flow guided self-supervised depth and camera pose estimation by an adaptive cross weighted loss to address the above limitations. Firstly, we explore to train the learnable occlusion mask fused optical flow network by an occlusion-aware photometric loss with the temporally suppleme…
3D scene flow estimation is a vital tool in perceiving our environment given depth or range sensors. Unlike optical flow, the data is usually sparse and in most cases partially occluded in between two temporal samplings. Here we propose a new scene flow architecture called OGSF-Net which tightly couples the learning for both flow and occlusions between frames. Their coupled symbiosis results in a more accurate prediction of flow in space. Unlike a traditional multi-action network, our unified approach is fused throughout the network, boosting performances for both occlusion detection and flow …
Monocular Depth Guided Occlusion-Aware Disparity Refinement via Semi-supervised Learning in Laparoscopic Images
School of Life Science and Technology, Harbin Institute of Technology（生命科学与技术学院，哈尔滨工业大学）；State Key Laboratory of Robotics and System, Harbin Institute of Technology（机器人与系统国家重点实验室，哈尔滨工业大学）
University of Washington（华盛顿大学）；ByteDance AI Lab（字节跳动人工智能实验室）
Recently, significant progress has been made in single-view depth estimation thanks to increasingly large and diverse depth datasets. However, these datasets are largely limited to specific application domains (e.g. indoor, autonomous driving) or static in-the-wild scenes due to hardware constraints or technical limitations of 3D reconstruction. In this paper, we introduce the first depth dataset DynOcc consisting of dynamic in-the-wild scenes. Our approach leverages the occlusion cues in these dynamic scenes to infer depth relationships between points of selected video frames. To achieve accu…
OccluGaussian: Occlusion-Aware Gaussian Splatting for Large Scene Reconstruction and Rendering
Huawei Noah’s Ark Lab（华为诺亚实验室）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；The University of Hong Kong（香港大学）；Huawei Embodied Intelligence Lab（华为具身智能实验室）
Gap Completion in Point Cloud Scene occluded by Vehicles using SGC-Net
Chair of Cartography and Visual Analytics, Technical University of Munich（慕尼黑工业大学制图与视觉分析主席）；Institute of Cartography and Geoinformatics, Leibniz University Hannover（汉诺威莱布尼茨大学制图与地理信息研究所）；Computer Vision Group, Technical University of Munich（慕尼黑工业大学计算机视觉组）
The University of Hong Kong（香港大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
We present a new learning-based method for multi-frame depth estimation from a color video, which is a fundamental problem in scene understanding, robot navigation or handheld 3D reconstruction. While recent learning-based methods estimate depth at high accuracy, 3D point clouds exported from their depth maps often fail to preserve important geometric feature (e.g., corners, edges, planes) of man-made scenes. Widely-used pixel-wise depth errors do not specifically penalize inconsistency on these features. These inaccuracies are particularly severe when subsequent depth reconstructions are accu…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Self-supervised monocular depth estimation, aiming to learn scene depths from single images in a self-supervised manner, has received much attention recently. In spite of recent efforts in this field, how to learn accurate scene depths and alleviate the negative influence of occlusions for self-supervised depth estimation, still remains an open problem. Addressing this problem, we firstly empirically analyze the effects of both the continuous and discrete depth constraints which are widely used in the training process of many existing works. Then inspired by the above empirical analysis, we pr…
DyCrowd: Towards Dynamic Crowd Reconstruction from a Large-scene Video
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学信息科学与技术全国重点研究院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学学院）
Representing and synthesizing novel views in real-world dynamic scenes from casual monocular videos is a long-standing problem. Existing solutions typically approach dynamic scenes by applying geometry techniques or utilizing temporal information between several adjacent frames without considering the underlying background distribution in the entire scene or the transmittance over the ray dimension, limiting their performance on static and occlusion areas. Our approach $\textbf{D}$istribution-$\textbf{D}$riven neural radiance fields offers high-quality view synthesis and a 3D solution to $\tex…
ShanghaiTech University（上海科技大学）；Chinese University of Hong Kong（香港中文大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉成像工程研究中心）
Real scans always miss partial geometries of objects due to the self-occlusions, external-occlusions, and limited sensor resolutions. Point cloud completion aims to refer the complete shapes for incomplete 3D scans of objects. Current deep learning-based approaches rely on large-scale complete shapes in the training process, which are usually obtained from synthetic datasets. It is not applicable for real-world scans due to the domain gap. In this paper, we propose a self-supervised point cloud completion method (TraPCC) for vehicles in real traffic scenes without any complete data. Based on t…
PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image
University of Tübingen, Zuse School ELIZA（图宾根大学Zuse学校ELIZA）；University of Tübingen, Tübingen AI Center（图宾根大学图宾根人工智能中心）；University of Tübingen, Tübingen AI Center, MPI for Informatics, SIC（图宾根大学图宾根人工智能中心、马克斯·普朗克信息研究所、SIC）
Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction
School of Engineering, Westlake University（西湖大学工学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
NVGS: Neural Visibility for Occlusion Culling in 3D Gaussian Splatting
Digital Future Lab, Flanders Make, Hasselt University（数字未来实验室，弗拉芒制造，哈塞尔特大学）；Computer Graphics Lab, TU Braunschweig（计算机图形学实验室，图林根工业大学）；Physics and Astronomy, University of New Mexico（物理与天文学，新墨西哥大学）
The University of New South Wales（新南威尔士大学）；Australian National University（澳大利亚国立大学）；North South University（北南大学）；University of Technology Sydney（悉尼科技大学）；Commonwealth Scientific and Industrial Research Organisation（联邦科学与工业研究组织）
Zero-shot learning on 3D point cloud data is a related underexplored problem compared to its 2D image counterpart. 3D data brings new challenges for ZSL due to the unavailability of robust pre-trained feature extraction models. To address this problem, we propose a prompt-guided 3D scene generation and supervision method that augments 3D data to learn the network better, exploring the complex interplay of seen and unseen objects. First, we merge point clouds of two 3D models in certain ways described by a prompt. The prompt acts like the annotation describing each 3D scene. Later, we perform c…
3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting
The Chinese University of Hong Kong（香港中文大学）；Stanford University（斯坦福大学）；Snap Inc.（Snap公司）；University of California Los Angeles（加利福尼亚大学洛杉矶分校）；ByteDance（字节跳动）
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipel…
We present a learning framework for reconstructing neural scene representations from a small number of unconstrained tourist photos. Since each image contains transient occluders, decomposing the static and transient components is necessary to construct radiance fields with such in-the-wild photographs where existing methods require a lot of training data. We introduce SF-NeRF, aiming to disentangle those two components with only a few images given, which exploits semantic information without any supervision. The proposed method contains an occlusion filtering module that predicts the transien…
Event-guided 3D Gaussian Splatting for Dynamic Human and Scene Reconstruction
College of Optical Science and Engineering, Zhejiang University, China（浙江大学光学工程学院）；School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；School of Artificial Intelligence and Robotics, Hunan University, China（湖南大学人工智能与机器人学院）
DFKI - German Research Center for Artificial Intelligence（德国人工智能研究中心）；BMW Group（宝马集团）
Motion estimation is one of the core challenges in computer vision. With traditional dual-frame approaches, occlusions and out-of-view motions are a limiting factor, especially in the context of environmental perception for vehicles due to the large (ego-) motion of objects. Our work proposes a novel data-driven approach for temporal fusion of scene flow estimates in a multi-frame setup to overcome the issue of occlusion. Contrary to most previous methods, we do not rely on a constant motion model, but instead learn a generic temporal relation of motion from data. In a second step, a neural ne…
PedHunter: Occlusion Robust Pedestrian Detector in Crowded Scenes
Institute of Electronics, Chinese Academy of Sciences（中国科学院电子学研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
OGMN: Occlusion-guided Multi-task Network for Object Detection in UAV Images
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Tracking and Telecommunication Technology（北京跟踪与通信技术研究所）
PWOC-3D: Deep Occlusion-Aware End-to-End Scene Flow Estimation
German Research Center for Artificial Intelligence - DFKI（德国人工智能研究中心（DFKI））；Birla Institute of Technology and Science - BITS Pilani（贝拉理工学院（BITS Pilani））；University of Kaiserslautern - TUK（凯泽斯劳滕大学（TUK））
