Inferring the depth of transparent or mirror (ToM) surfaces represents a hard challenge for either sensors, algorithms, or deep networks. We propose a simple pipeline for learning to estimate depth properly for such surfaces with neural networks, without requiring any ground-truth annotation. We unveil how to obtain reliable pseudo labels by in-painting ToM objects in images and processing them with a monocular depth estimation model. These labels can be used to fine-tune existing monocular or stereo networks, to let them learn how to deal with ToM surfaces. Experimental results on the Booster dataset show the dramatic improvements enabled by our remarkably simple proposal.
Despite recent progress in depth sensing and 3D reconstruction, mirror surfaces are a significant source of errors. To address this problem, we create the Mirror3D dataset: a 3D mirror plane dataset based on three RGBD datasets (Matterport3D, NYUv2 and ScanNet) containing 7,011 mirror instance masks and 3D planes. We then develop Mirror3DNet: a module that refines raw sensor depth or estimated depth to correct errors on mirror surfaces. Our key idea is to estimate the 3D mirror plane based on RGB input and surrounding depth context, and use this estimate to directly regress mirror surface dept…
Neural networks have shown great success in extracting geometric information from color images. Especially, monocular depth estimation networks are increasingly reliable in real-world scenes. In this work we investigate the applicability of such monocular depth estimation networks to semi-transparent volume rendered images. As depth is notoriously difficult to define in a volumetric scene without clearly defined surfaces, we consider different depth computations that have emerged in practice, and compare state-of-the-art monocular depth estimation approaches for these different interpretations…
Rethinking Transparent Object Grasping: Depth Completion with Monocular Depth Estimation and Instance Mask
State Key Laboratory of Robotics and System at Harbin Institute of Technology（哈尔滨工业大学机器人系统国家重点实验室）；Lanzhou University of Technology（兰州理工大学）；Department of Computer Science（计算机科学系）；University of Liverpool（利物浦大学）
Commercial depth sensors usually generate noisy and missing depths, especially on specular and transparent objects, which poses critical issues to downstream depth or point cloud-based tasks. To mitigate this problem, we propose a powerful RGBD fusion network, SwinDRNet, for depth restoration. We further propose Domain Randomization-Enhanced Depth Simulation (DREDS) approach to simulate an active stereo depth system using physically based rendering and generate a large-scale synthetic dataset that contains 130K photorealistic RGB images along with their simulated depths carrying realistic sens…
Multi-Label Stereo Matching for Transparent Scene Depth Estimation
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, China（北京智能信息科技重点实验室，计算机科学与技术学院，北京理工大学，中国）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, China（广东机器感知与智能计算实验室，深圳MSU-BIT大学，中国）
University of Montreal（蒙特利尔大学）；University of Science and Technology（科技大学）
Reconstructing 3D object models is playing an important role in many applications in the field of computer vision. Instead of employing a collection of cameras and/or sensors as in many studies, this paper proposes a simple way to build a cheaper system for 3D reconstruction using only one depth camera and 2 or more mirrors. Each mirror is equivalently considered as a depth camera at another viewpoint. Since all scene data are provided by only one depth sensor, our approach can be applied to moving objects and does not require any synchronization protocol as with a set of cameras. Some experim…
University of Montreal（蒙特利尔大学）；University of Science and Technology（科技大学）
This paper proposes a reliable approach for human gait symmetry assessment using a depth camera and two mirrors. The input of our system is a sequence of 3D point clouds which are formed from a setup including a Time-of-Flight (ToF) depth camera and two mirrors. A cylindrical histogram is estimated for describing the posture in each point cloud. The sequence of such histograms is then separated into two sequences of sub-histograms representing two half-bodies. A cross-correlation technique is finally applied to provide values describing gait symmetry indices. The evaluation was performed on 9 …
University of Science and Technology of China（中国科学技术大学）；Huazhong University of Science and Technology（华中科技大学）
Transparent objects are widely used in industrial automation and daily life. However, robust visual recognition and perception of transparent objects have always been a major challenge. Currently, most commercial-grade depth cameras are still not good at sensing the surfaces of transparent objects due to the refraction and reflection of light. In this work, we present a transformer-based transparent object depth estimation approach from a single RGB-D input. We observe that the global characteristics of the transformer make it easier to extract contextual information to perform depth estimatio…
Monocular Depth Estimation and Segmentation for Transparent Object with Iterative Semantic and Geometric Fusion
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所）；School of Information Science and Technology, Shijiazhuang Tiedao University（石家庄铁道大学信息科学与技术学院）；D-Robotics（杭州宇树科技有限公司）
Majority of the perception methods in robotics require depth information provided by RGB-D cameras. However, standard 3D sensors fail to capture depth of transparent objects due to refraction and absorption of light. In this paper, we introduce a new approach for depth completion of transparent objects from a single RGB-D image. Key to our approach is a local implicit neural representation built on ray-voxel pairs that allows our method to generalize to unseen objects and achieve fast inference speed. Based on this representation, we present a novel framework that can complete missing depth gi…
GAA-TSO: Geometry-Aware Assisted Depth Completion for Transparent and Specular Objects
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization（工业智能与系统优化国家前沿科学中心）
ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg（汉堡大学信息学院）；Agile Robots SE（敏捷机器人公司）；Technical University of Munich（慕尼黑技术大学）；Mohamed Bin Zayed University of Artificial Intelligence (MBUZAI)（Mohamed Bin Zayed人工智能大学）
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Nvidia（英伟达）
The basis of many object manipulation algorithms is RGB-D input. Yet, commodity RGB-D sensors can only provide distorted depth maps for a wide range of transparent objects due light refraction and absorption. To tackle the perception challenges posed by transparent objects, we propose TranspareNet, a joint point cloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels. To address the shortcomings of existing transparent object data collection schemes in literatu…
The ability to grasp and manipulate transparent objects is a major challenge for robots. Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of such objects. We propose using neural radiance fields (NeRF) to detect, localize, and infer the geometry of transparent objects with sufficient accuracy to find and grasp them securely. We leverage NeRF's view-independent learned density, place lights to increase specular reflections, and perform a transparency-aware depth-rendering that we feed into the Dex-Net grasp planner. We show how additional lights create sp…
SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping
Beijing University of Posts and Telecommunications（北京邮电大学）；Center on Frontiers of Computing Studies, Peking University（北京大学前沿计算研究中心）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Automation and Control Institute (ACIN)（自动化与控制研究所）；TU Wien（维也纳工业大学）
Transparent objects are common in day-to-day life and hence find many applications that require robot grasping. Many solutions toward object grasping exist for non-transparent objects. However, due to the unique visual properties of transparent objects, standard 3D sensors produce noisy or distorted measurements. Modern approaches tackle this problem by either refining the noisy depth measurements or using some intermediate representation of the depth. Towards this, we study deep learning 6D pose estimation from RGB images only for transparent object grasping. To train and test the suitability…
ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera
Samsung R&D Institute China-Beijing（三星中国北京研发中心）；CFCS, School of Computer Science, Peking University（计算机科学系，北京大学）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）
Transparent objects are a common part of everyday life, yet they possess unique visual properties that make them incredibly difficult for standard 3D sensors to produce accurate depth estimates for. In many cases, they often appear as noisy or distorted approximations of the surfaces that lie behind them. To address these challenges, we present ClearGrasp -- a deep learning approach for estimating accurate 3D geometry of transparent objects from a single RGB-D image for robotic manipulation. Given a single RGB-D image of transparent objects, ClearGrasp uses deep convolutional networks to infer…
TransBiolab: A Real-World Multi-View Dataset of Cluttered Transparent Biomedical Objects
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；College of Design and Innovation, Tongji University（同济大学设计创意学院）；Shanghai Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能无人系统研究院）；School of Software and Engineering, Huazhong University of Science and Technology（华中科技大学软件学院）
Self-Supervised Learning for Transparent Object Depth Completion Using Depth from Non-Transparent Objects
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences（长春光学精密机械与物理研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Fudan University（复旦大学）
We present a novel algorithm for self-supervised monocular depth completion. Our approach is based on training a neural network that requires only sparse depth measurements and corresponding monocular video sequences without dense depth labels. Our self-supervised algorithm is designed for challenging indoor environments with textureless regions, glossy and transparent surface, non-Lambertian surfaces, moving people, longer and diverse depth ranges and scenes captured by complex ego-motions. Our novel architecture leverages both deep stacks of sparse convolution blocks to extract sparse depth …
TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians
State Key Lab for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；TMCC, College of Computer Science, Nankai University（天津大学计算机学院）；State Key Lab of CAD & CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；Institute of Hangzhou Holographic Intelligent Technology（杭州全息智能技术研究院）
TDCNet: Transparent Objects Depth Completion with CNN-Transformer Dual-Branch Parallel Network
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences（中国科学院长春光学精密机械与物理研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Institute of Technology（哈尔滨工业大学）；College of Physics, Northeast Normal University（东北师范大学物理学院）
The Chinese University of Hong Kong（香港中文大学）；University of California, Berkeley（加州大学伯克利分校）；Dyson Robot Learning Lab（戴森机器人学习实验室）；Hong Kong Centre for Logistics Robotics（香港物流机器人中心）
Most existing methods for category-level pose estimation rely on object point clouds. However, when considering transparent objects, depth cameras are usually not able to capture meaningful data, resulting in point clouds with severe artifacts. Without a high-quality point cloud, existing methods are not applicable to challenging transparent objects. To tackle this problem, we present StereoPose, a novel stereo image framework for category-level object pose estimation, ideally suited for transparent objects. For a robust estimation from pure stereo images, we develop a pipeline that decouples …
University of Montreal（蒙特利尔大学）；University of Science and Technology（科技大学）
This paper presents an initial work on assessment of gait normality in which the human body motion is represented by a sequence of enhanced depth maps. The input data is provided by a system consisting of a Time-of-Flight (ToF) depth camera and two mirrors. This approach proposes two feature types to describe characteristics of localized points of interest and the level of posture symmetry. These two features are processed on a sequence of enhanced depth maps with the support of a sliding window to provide two corresponding scores. The gait assessment is finally performed based on a weighted c…
Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）；AI Institute（人工智能研究院）
Transparent objects are common in our daily life and frequently handled in the automated production line. Robust vision-based robotic grasping and manipulation for these objects would be beneficial for automation. However, the majority of current grasping algorithms would fail in this case since they heavily rely on the depth image, while ordinary depth sensors usually fail to produce accurate depth information for transparent objects owing to the reflection and refraction of light. In this work, we address this issue by contributing a large-scale real-world dataset for transparent object dept…
