Transparent objects are common in daily life. However, depth sensing for transparent objects remains a challenging problem. While learning-based methods can leverage shape priors to improve the sensing quality, the labor-intensive data collection in the real world and the sim-to-real domain gap restrict these methods' scalability. In this paper, we propose a method to finetune a stereo network with sparse depth labels automatically collected using a probing system with tactile feedback. We present a novel utility function to evaluate the benefit of touches. By approximating and optimizing the utility function, we can optimize the probing locations given a fixed touching budget to better improve the network's performance on real objects. We further combine tactile depth supervision with a confidence-based regularization to prevent over-fitting during finetuning. To evaluate the effectiveness of our method, we construct a real-world dataset including both diffuse and transparent objects. Experimental results on this dataset show that our method can significantly improve real-world depth sensing accuracy, especially for transparent objects.
TDCNet: Transparent Objects Depth Completion with CNN-Transformer Dual-Branch Parallel Network
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences(中国科学院长春光学精密机械与物理研究所);University of Chinese Academy of Sciences(中国科学院大学);Harbin Institute of Technology(哈尔滨工业大学);College of Physics, Northeast Normal University(东北师范大学物理学院)
Trans2Occ: Voxel Occupancy Estimation and Grasp for Transparent Objects from Simulation to Reality
Shanghai AI Laboratory(上海人工智能实验室);SUSTech(南方科技大学);CUHK(香港中文大学);Harbin Institute of Technology(哈尔滨工业大学);University of Oxford(牛津大学);Beihang University(北京航空航天大学);Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学)
SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping
Beijing University of Posts and Telecommunications(北京邮电大学);Center on Frontiers of Computing Studies, Peking University(北京大学前沿计算研究中心);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
D3RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation
CFCS;School of Computer Science, Peking University(北京大学计算机学院);Galbot;University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);University of Chinese Academy of Sciences(中国科学院大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
AISPO: Enhancing Depth Reliability for Robotic Manipulation of Non-Lambertian Objects via Affine-Invariant Shape Prior
The Hong Kong University of Science & Technology(香港科技大学);The University of Hong Kong(香港大学);Southern University of Science & Technology(南方科技大学);Zhejiang University(浙江大学);LimX Dynamics