In the robotic industry, specular and textureless metallic components are ubiquitous. The 6D pose estimation of such objects with only a monocular RGB camera is difficult because of the absence of rich texture features. Furthermore, the appearance of specularity heavily depends on the camera viewpoint and environmental light conditions making traditional methods, like template matching, fail. In the last 30 years, pose estimation of the specular object has been a consistent challenge, and most related works require massive knowledge modeling effort for light setups, environment, or the object surface. On the other hand, recent works exhibit the feasibility of 6D pose estimation on a monocular camera with convolutional neural networks(CNNs) however they mostly use opaque objects for evaluation. This paper provides a data-driven solution to estimate the 6D pose of specular objects for grasping them, proposes a cost function for handling symmetry, and demonstrates experimental results showing the system's feasibility.
SLAMcore Ltd（SLAMcore有限公司）；Université Paris Saclay（巴黎萨克雷大学）；CEA（法国原子能和替代能源委员会）；LIST（（法国原子能委员会）实验室集成系统与技术研究所）；CNRS（法国国家科学研究中心）；UCA（克莱蒙奥弗涅大学）；CHU（克莱蒙费朗大学医院）
Specularity prediction is essential to many computer vision applications, giving important visual cues usable in Augmented Reality (AR), Simultaneous Localisation and Mapping (SLAM), 3D reconstruction and material modeling. However, it is a challenging task requiring numerous information from the scene including the camera pose, the geometry of the scene, the light sources and the material properties. Our previous work addressed this task by creating an explicit model using an ellipsoid whose projection fits the specularity image contours for a given camera pose. These ellipsoid-based approach…
Commercial depth sensors usually generate noisy and missing depths, especially on specular and transparent objects, which poses critical issues to downstream depth or point cloud-based tasks. To mitigate this problem, we propose a powerful RGBD fusion network, SwinDRNet, for depth restoration. We further propose Domain Randomization-Enhanced Depth Simulation (DREDS) approach to simulate an active stereo depth system using physically based rendering and generate a large-scale synthetic dataset that contains 130K photorealistic RGB images along with their simulated depths carrying realistic sens…
SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping
Beijing University of Posts and Telecommunications（北京邮电大学）；Center on Frontiers of Computing Studies, Peking University（北京大学前沿计算研究中心）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
GAA-TSO: Geometry-Aware Assisted Depth Completion for Transparent and Specular Objects
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization（工业智能与系统优化国家前沿科学中心）
Dynamic Neural Radiance Field (NeRF) is a powerful algorithm capable of rendering photo-realistic novel view images from a monocular RGB video of a dynamic scene. Although it warps moving points across frames from the observation spaces to a common canonical space for rendering, dynamic NeRF does not model the change of the reflected color during the warping. As a result, this approach often fails drastically on challenging specular objects in motion. We address this limitation by reformulating the neural radiance field function to be conditioned on surface position and orientation in the obse…
Chongqing University（重庆大学）；MAIS, Institute of Automation, Chinese Academy of Sciences and UCAS（自动化研究所，中国科学院，UCAS）；King Abdullah University of Science and Technology (KAUST)（卡塔尔科学与技术大学）
In advanced mission concepts with high levels of autonomy, spacecraft need to internally model the pose and shape of nearby orbiting objects. Recent works in neural scene representations show promising results for inferring generic three-dimensional scenes from optical images. Neural Radiance Fields (NeRF) have shown success in rendering highly specular surfaces using a large number of images and their pose. More recently, Generative Radiance Fields (GRAF) achieved full volumetric reconstruction of a scene from unposed images only, thanks to the use of an adversarial framework to train a NeRF.…
Laboratoire Bordelais de Recherche Informatique, Université de Bordeaux（波尔多大学波尔多计算机研究实验室）；Technische Universät München（慕尼黑工业大学）；Siemens AG（西门子股份公司）
Objects with symmetries are common in our daily life and in industrial contexts, but are often ignored in the recent literature on 6D pose estimation from images. In this paper, we study in an analytical way the link between the symmetries of a 3D object and its appearance in images. We explain why symmetrical objects can be a challenge when training machine learning algorithms that aim at estimating their 6D pose from images. We propose an efficient and simple solution that relies on the normalization of the pose rotation. Our approach is general and can be used with any 6D pose estimation al…
Object pose estimation is a necessary prerequisite for autonomous robotic manipulation, but the presence of symmetry increases the complexity of the pose estimation task. Existing methods for object pose estimation output a single 6D pose. Thus, they lack the ability to reason about symmetries. Lately, modeling object orientation as a non-parametric probability distribution on the SO(3) manifold by neural networks has shown impressive results. However, acquiring large-scale datasets to train pose estimation models remains a bottleneck. To address this limitation, we introduce an automatic pose…
The point pair feature (PPF) is widely used for 6D pose estimation. In this paper, we propose an efficient 6D pose estimation method based on the PPF framework. We introduce a well-targeted down-sampling strategy that focuses more on edge area for efficient feature extraction of complex geometry. A pose hypothesis validation approach is proposed to resolve the symmetric ambiguity by calculating edge matching degree. We perform evaluations on two challenging datasets and one real-world collected dataset, demonstrating the superiority of our method on pose estimation of geometrically complex, oc…
PCKRF: Point Cloud Completion and Keypoint Refinement With Fusion Data for 6D Pose Estimation
Faculty of Information Technology, Beijing University of Technology（北京工业大学信息学部）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Beijing Institute of Technology（北京理工大学）；Shanghai Jiao Tong University（上海交通大学）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consis…
Cross-Modal RGB-D Fusion Transformer for 6D Pose Estimation of Non-Cooperative Spacecraft with Stereo-Derived Depth
School of Physics, Northeast Normal University（东北师范大学物理学院）；Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Science（长春光学精密机械与物理研究所）
TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians
State Key Lab for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；TMCC, College of Computer Science, Nankai University（天津大学计算机学院）；State Key Lab of CAD & CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；Institute of Hangzhou Holographic Intelligent Technology（杭州全息智能技术研究院）
Self-supervised learning of depth has been a highly studied topic of research as it alleviates the requirement of having ground truth annotations for predicting depth. Depth is learnt as an intermediate solution to the task of view synthesis, utilising warped photometric consistency. Although it gives good results when trained using stereo data, the predicted depth is still sensitive to noise, illumination changes and specular reflections. Also, occlusion can be tackled better by learning depth from a single camera. We propose ADAA, utilising depth augmentation as depth supervision for learnin…
Dental3R: Geometry-Aware Pairing for Intraoral 3D Reconstruction from Sparse-View Photographs
School of AI and Advanced Computing, Xi'an Jiaotong-Liverpool University, China（人工智能与先进计算学院，西安交通大学利物浦大学，中国）；School of Electrical Engineering, Electronics and Computer Science, University of Liverpool, United Kingdom（电子工程与计算机科学学院，利物浦大学，英国）；School of Advanced Technology, Xi'an Jiaotong-Liverpool University, China（先进技术学院，西安交通大学利物浦大学，中国）；School of Physical Sciences, University of Liverpool, Liverpool, United Kingdom（物理科学学院，利物浦大学，英国）；School of Mathematics and Physics, Xi'an Jiaotong-Liverpool University, China（数学与物理学院，西安交通大学利物浦大学，中国）；Institute of Artificial Intelligence Innovation, Zhejiang University of Technology, China（人工智能创新研究院，浙江工业大学，中国）
Transparent objects present multiple distinct challenges to visual perception systems. First, their lack of distinguishing visual features makes transparent objects harder to detect and localize than opaque objects. Even humans find certain transparent surfaces with little specular reflection or refraction, e.g. glass doors, difficult to perceive. A second challenge is that common depth sensors typically used for opaque object perception cannot obtain accurate depth measurements on transparent objects due to their unique reflective properties. Stemming from these challenges, we observe that tr…
ColonAdapter: Geometry Estimation Through Foundation Model Adaptation for Colonoscopy
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Vetex Lab（Vetex实验室）；Southeast University - Monash University Joint Graduate School（东南大学-墨尔本大学联合研究生院）；Monash Suzhou Research Institute（墨尔本大学苏州研究院）；Faculty of Information Technology, Monash University（墨尔本大学信息科技学院）
We present a large-scale stereo RGB image object pose estimation dataset named the $\textbf{StereOBJ-1M}$ dataset. The dataset is designed to address challenging cases such as object transparency, translucency, and specular reflection, in addition to the common challenges of occlusion, symmetry, and variations in illumination and environments. In order to collect data of sufficient scale for modern deep learning models, we propose a novel method for efficiently annotating pose data in a multi-view fashion that allows data capturing in complex and flexible environments. Fully annotated with 6D …
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；Robotics Institute（机器人研究所）
Depth acquisition with the active stereo camera is a challenging task for highly reflective objects. When setup permits, multi-view fusion can provide increased levels of depth completion. However, due to the slow acquisition speed of high-end active stereo cameras, collecting a large number of viewpoints for a single scene is generally not practical. In this work, we propose a next-best-view framework to strategically select camera viewpoints for completing depth data on reflective objects. In particular, we explicitly model the specular reflection of reflective surfaces based on the Phong re…
