Recovering the 3D shape of transparent objects using a small number of unconstrained natural images is an ill-posed problem. Complex light paths induced by refraction and reflection have prevented both traditional and deep multiview stereo from solving this challenge. We propose a physically-based network to recover 3D shape of transparent objects using a few images acquired with a mobile phone camera, under a known but arbitrary environment map. Our novel contributions include a normal representation that enables the network to model complex light transport through local computation, a rendering layer that models refractions and reflections, a cost volume specifically designed for normal refinement of transparent shapes and a feature mapping based on predicted normals for 3D point cloud reconstruction. We render a synthetic dataset to encourage the model to learn refractive light transport across different views. Our experiments show successful recovery of high-quality 3D geometry for complex transparent shapes using as few as 5-12 natural images. Code and data are publicly released.
The Australian National University（澳大利亚国立大学）；Data61, CSIRO（联邦科学与工业研究组织Data61研究所）
In this paper, we define a new problem of recovering the 3D geometry of an object confined in a transparent enclosure. We also propose a novel method for solving this challenging problem. Transparent enclosures pose challenges of multiple light reflections and refractions at the interface between different propagation media e.g. air or glass. These multiple reflections and refractions cause serious image distortions which invalidate the single viewpoint assumption. Hence the 3D geometry of such objects cannot be reliably reconstructed using existing methods, such as traditional structure from …
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Nvidia（英伟达）
The basis of many object manipulation algorithms is RGB-D input. Yet, commodity RGB-D sensors can only provide distorted depth maps for a wide range of transparent objects due light refraction and absorption. To tackle the perception challenges posed by transparent objects, we propose TranspareNet, a joint point cloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels. To address the shortcomings of existing transparent object data collection schemes in literatu…
GRAR: Glass-induced Reflection Artifact Removal in LiDAR Point Clouds
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；School of Design, Hunan University（湖南大学设计学院）；School of Finance and Statistics, Hunan University（湖南大学金融与统计学院）
Australian Centre for Robotic Vision, Queensland University of Technology (QUT)（昆士兰科技大学澳大利亚机器人视觉中心）；Sydney Institute for Robotics and Intelligent Systems, University of Sydney（悉尼大学悉尼机器人与智能系统研究所）
Curved refractive objects are common in the human environment, and have a complex visual appearance that can cause robotic vision algorithms to fail. Light-field cameras allow us to address this challenge by capturing the view-dependent appearance of such objects in a single exposure. We propose a novel image feature for light fields that detects and describes the patterns of light refracted through curved transparent objects. We derive characteristic points based on these features allowing them to be used in place of conventional 2D features. Using our features, we demonstrate improved struct…
Transparent objects are prevalent across many environments of interest for dexterous robotic manipulation. Such transparent material leads to considerable uncertainty for robot perception and manipulation, and remains an open challenge for robotics. This problem is exacerbated when multiple transparent objects cluster into piles of clutter. In household environments, for example, it is common to encounter piles of glassware in kitchens, dining rooms, and reception areas, which are essentially invisible to modern robots. We present the GlassLoc algorithm for grasp pose detection of transparent …
With the recent growth in computer vision applications, the question of how fair and unbiased they are has yet to be explored. There is abundant evidence that the bias present in training data is reflected in the models, or even amplified. Many previous methods for image dataset de-biasing, including models based on augmenting datasets, are computationally expensive to implement. In this study, we present a fast and effective model to de-bias an image dataset through reconstruction and minimizing the statistical dependence between intended variables. Our architecture includes a U-net to recons…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Dalian University of Technology（大连理工大学）；Nanjing University（南京大学）；Lingnan University（岭南大学）
Most of the existing object detection methods generate poor glass detection results, due to the fact that the transparent glass shares the same appearance with arbitrary objects behind it in an image. Different from traditional deep learning-based wisdoms that simply use the object boundary as auxiliary supervision, we exploit label decoupling to decompose the original labeled ground-truth (GT) map into an interior-diffusion map and a boundary-diffusion map. The GT map in collaboration with the two newly generated maps breaks the imbalanced distribution of the object boundary, leading to impro…
Through the Looking Glass: A Dual Perspective on Weakly-Supervised Few-Shot Segmentation
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
Detection of boundaries of materials stored in transparent vessels is essential for identifying properties such as liquid level and phase boundaries, which are vital for controlling numerous processes in the industry and chemistry laboratory. This work presents a computer vision method for identifying the boundary of materials in transparent vessels using the graph-cut algorithm. The method receives an image of a transparent vessel containing a material and the contour of the vessel in the image. The boundary of the material in the vessel is found by the graph cut method. In general the method…
Real-Time Glass Detection and Reprojection using Sensor Fusion Onboard Aerial Robots
ARL DCIST CRA；NSF Grant CCR-2112665（NSF CCR-2112665）；NSF Graduate Research Fellowship Grant No. DGE-1845298（NSF DGE-1845298）；IoT4Ag Engineering Research Center
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Sciences（中国科学院）
Image matting refers to predicting the alpha values of unknown foreground areas from natural images. Prior methods have focused on propagating alpha values from known to unknown regions. However, not all natural images have a specifically known foreground. Images of transparent objects, like glass, smoke, web, etc., have less or no known foreground. In this paper, we propose a Transformer-based network, TransMatting, to model transparent objects with a big receptive field. Specifically, we redesign the trimap as three learnable tri-tokens for introducing advanced semantic features into the sel…
Influence of Water Droplet Contamination for Transparency Segmentation
Technical University of Darmstadt（达姆施塔特工业大学）；Fraunhofer Institute for Computer Graphics Research IGD（弗劳恩霍夫计算机图形学研究所）；CGV Institute, Graz University of Technology（格拉茨工业大学CGV研究所）
University of Alberta（阿尔伯塔大学）；Snapchat（Snapchat公司）；University of Manitoba（曼尼托巴大学）
This paper proposes a new glass segmentation method utilizing paired RGB and thermal images. Due to the large difference between the transmission property of visible light and that of the thermal energy through the glass where most glass is transparent to the visible light but opaque to thermal energy, glass regions of a scene are made more distinguishable with a pair of RGB and thermal images than solely with an RGB image. To exploit such a unique property, we propose a neural network architecture that effectively combines an RGB-thermal image pair with a new multi-modal fusion module based o…
Computer Vision for Human-Computer Interaction Lab（人机交互计算机视觉实验室）；Center for Digital Accessibility and Assistive Technology, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院数字无障碍与辅助技术中心）
Transparent objects, such as glass walls and doors, constitute architectural obstacles hindering the mobility of people with low vision or blindness. For instance, the open space behind glass doors is inaccessible, unless it is correctly perceived and interacted with. However, traditional assistive technologies rarely cover the segmentation of these safety-critical transparent objects. In this paper, we build a wearable system with a novel dual-head Transformer for Transparency (Trans4Trans) perception model, which can segment general- and transparent objects. The two dense segmentation result…
Transparent objects present multiple distinct challenges to visual perception systems. First, their lack of distinguishing visual features makes transparent objects harder to detect and localize than opaque objects. Even humans find certain transparent surfaces with little specular reflection or refraction, e.g. glass doors, difficult to perceive. A second challenge is that common depth sensors typically used for opaque object perception cannot obtain accurate depth measurements on transparent objects due to their unique reflective properties. Stemming from these challenges, we observe that tr…
Guangzhou Huya Information Technology Co Ltd（广州虎牙信息科技有限公司）；The University of Hong Kong（香港大学）；National University of Singapore（新加坡国立大学）；Tencent Youtu（腾讯优图）
Detecting transparent objects in natural scenes is challenging due to the low contrast in texture, brightness and colors. Recent deep-learning-based works reveal that it is effective to leverage boundaries for transparent object detection (TOD). However, these methods usually encounter boundary-related imbalance problem, leading to limited generation capability. Detailly, a kind of boundaries in the background, which share the same characteristics with boundaries of transparent objects but have much smaller amounts, usually hurt the performance. To conquer the boundary-related imbalance proble…
The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）；Nanjing University（南京大学）；The University of Adelaide（阿德莱德大学）
Transparent objects such as windows and bottles made by glass widely exist in the real world. Segmenting transparent objects is challenging because these objects have diverse appearance inherited from the image background, making them had similar appearance with their surroundings. Besides the technical difficulty of this task, only a few previous datasets were specially designed and collected to explore this task and most of the existing datasets have major drawbacks. They either possess limited sample size such as merely a thousand of images without manual annotations, or they generate all i…
Technion – Israel Institute of Technology（以色列理工学院）
The recognition of materials and objects inside transparent containers using computer vision has a wide range of applications, ranging from industrial bottles filling to the automation of chemistry laboratory. One of the main challenges in such recognition is the ability to distinguish between image features resulting from the vessels surface and image features resulting from the material inside the vessel. Reflections and the functional parts of a vessels surface can create strong edges that can be mistakenly identified as corresponding to the vessel contents, and cause recognition errors. Th…
