Head Pose Estimation and 3D Neural Surface Reconstruction via Monocular Camera in situ for Navigation and Safe Insertion into Natural Openings
The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong（香港中文大学舜兴高等工程研究所）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）
As the significance of simulation in medical care and intervention continues to grow, it is anticipated that a simplified and low-cost platform can be set up to execute personalized diagnoses and treatments. 3D Slicer can not only perform medical image analysis and visualization but can also provide surgical navigation and surgical planning functions. In this paper, we have chosen 3D Slicer as our base platform and monocular cameras are used as sensors. Then, We used the neural radiance fields (NeRF) algorithm to complete the 3D model reconstruction of the human head. We compared the accuracy of the NeRF algorithm in generating 3D human head scenes and utilized the MarchingCube algorithm to generate corresponding 3D mesh models. The individual's head pose, obtained through single-camera vision, is transmitted in real-time to the scene created within 3D Slicer. The demonstrations presented in this paper include real-time synchronization of transformations between the human head model in the 3D Slicer scene and the detected head posture. Additionally, we tested a scene where a tool, marked with an ArUco Maker tracked by a single camera, synchronously points to the real-time transformation of the head posture. These demos indicate that our methodology can provide a feasible real-time simulation platform for nasopharyngeal swab collection or intubation.
3D Copy-Paste: Physically Plausible Object Insertion for Monocular 3D Detection
Stanford University（斯坦福大学）；University of Southern California（南加州大学）；Bosch Research North America（博世北美研究院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心（BCAI））
Tsinghua University（清华大学）；City University of Hong Kong（香港城市大学）
In this paper, we propose a scale-aware method for inserting virtual objects with proper sizes into monocular videos. To tackle the scale ambiguity problem of geometry recovery from monocular videos, we estimate the global scale objects in a video with a Bayesian approach incorporating the size priors of objects, where the scene objects sizes should strictly conform to the same global scale and the possibilities of global scales are maximized according to the size distribution of object categories. To do so, we propose a dataset of sizes of object categories: Metric-Tree, a hierarchical repres…
Introducing a Class-Aware Metric for Monocular Depth Estimation: An Automotive Perspective
Dr. Ing. h.c. F. Porsche AG（保时捷股份公司）；Ulm University（乌尔姆大学）；Porsche Engineering Group GmbH（保时捷工程集团有限公司）；Stuttgart Media University（斯图加特媒体大学）；University of Freiburg（弗莱堡大学）
RePoseD: Efficient Relative Pose Estimation With Known Depth Information
Czech Technical University in Prague（布拉格捷克技术大学）；Comenius University in Bratislava（布拉迪斯拉发夸美纽斯大学）；Nankai University（南开大学）；Czech Institute of Informatics, Robotics and Cybernetics（捷克信息学、机器人学与控制论研究所）
University of California San Diego（加利福尼亚大学圣迭戈分校）；Adobe Research（奥多比研究院）
Most 3D reconstruction methods may only recover scene properties up to a global scale ambiguity. We present a novel approach to single view metrology that can recover the absolute scale of a scene represented by 3D heights of objects or camera height above the ground as well as camera parameters of orientation and field of view, using just a monocular image acquired in unconstrained condition. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object he…
Autonomous robot manipulation involves estimating the translation and orientation of the object to be manipulated as a 6-degree-of-freedom (6D) pose. Methods using RGB-D data have shown great success in solving this problem. However, there are situations where cost constraints or the working environment may limit the use of RGB-D sensors. When limited to monocular camera data only, the problem of object pose estimation is very challenging. In this work, we introduce a novel method called SilhoNet that predicts 6D object pose from monocular images. We use a Convolutional Neural Network (CNN) pi…
We present a processing technique for a robust reconstruction of motion properties for single points in large scale, dynamic environments. We assume that the acquisition camera is moving and that there are other independently moving agents in a large environment, like road scenarios. The separation of direction and magnitude of the reconstructed motion allows for robust reconstruction of the dynamic state of the objects in situations, where conventional binocular systems fail due to a small signal (disparity) from the images due to a constant detection error, and where structure from motion ap…
Inria（法国国家信息与自动化研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学机器人与工业信息技术研究所）
Recent literature addressed the monocular 3D pose estimation task very satisfactorily. In these studies, different persons are usually treated as independent pose instances to estimate. However, in many every-day situations, people are interacting, and the pose of an individual depends on the pose of his/her interactees. In this paper, we investigate how to exploit this dependency to enhance current - and possibly future - deep networks for 3D monocular pose estimation. Our pose interacting network, or PI-Net, inputs the initial pose estimates of a variable number of interactees into a recurre…
Tata Consultancy Services（塔塔咨询服务公司）；Indian Institute of Technology Guwahati（古瓦哈提印度理工学院）
This paper looks into the problem of pedestrian tracking using a monocular, potentially moving, uncalibrated camera. The pedestrians are located in each frame using a standard human detector, which are then tracked in subsequent frames. This is a challenging problem as one has to deal with complex situations like changing background, partial or full occlusion and camera motion. In order to carry out successful tracking, it is necessary to resolve associations between the detected windows in the current frame with those obtained from the previous frame. Compared to methods that use temporal win…
Visual SLAM shows significant progress in recent years due to high attention from vision community but still, challenges remain for low-textured environments. Feature based visual SLAMs do not produce reliable camera and structure estimates due to insufficient features in a low-textured environment. Moreover, existing visual SLAMs produce partial reconstruction when the number of 3D-2D correspondences is insufficient for incremental camera estimation using bundle adjustment. This paper presents Edge SLAM, a feature based monocular visual SLAM which mitigates the above mentioned problems. Our p…
AASS research center, Örebro University, Sweden（奥雷布罗大学AASS研究中心）；Czech Technical University in Prague（布拉格捷克技术大学）；Technical University of Munich, Chair: Perception for Intelligent Systems（慕尼黑技术大学感知智能系统教研室）
Hazard detection and avoidance is a key technology for future robotic small body sample return and lander missions. Current state-of-the-practice methods rely on high-fidelity, a priori terrain maps, which require extensive human-in-the-loop verification and expensive reconnaissance campaigns to resolve mapping uncertainties. We propose a novel safety mapping paradigm that leverages deep semantic segmentation techniques to predict landing safety directly from a single monocular image, thus reducing reliance on high-fidelity, a priori data products. We demonstrate precise and accurate safety ma…
We propose an approach to estimate arm and hand dynamics from monocular video by utilizing the relationship between arm and hand. Although monocular full human motion capture technologies have made great progress in recent years, recovering accurate and plausible arm twists and hand gestures from in-the-wild videos still remains a challenge. To solve this problem, our solution is proposed based on the fact that arm poses and hand gestures are highly correlated in most real situations. To fully exploit arm-hand correlation as well as inter-frame information, we carefully design a Spatial-Tempor…
RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions
National University of Singapore（新加坡国立大学）；Huazhong University of Science and Technology（华中科技大学）；Carnegie Mellon University（卡内基梅隆大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；IPAL, CNRS IRL 2955（法国国家科学研究中心国际联合实验室2955（IPAL））；CerCo, CNRS UMR 5549, Université Toulouse III（图卢兹第三大学法国国家科学研究中心混合研究单位5549（CerCo））
Endoscopic Depth Estimation Based on Deep Learning: A Survey
Beijing Information Science and Technology University（北京信息科技大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（南方科技大学可信自主系统研究院）；The Third Hospital of Xiamen（厦门市第三医院）；Fujian Institute of Geriatrics（福建省老年医学研究所）；Fujian Medical University Union Hospital（福建医科大学附属协和医院）；Beijing Tiromu Medical Technology Co., Ltd.（北京提罗木医疗科技有限公司）；National University of Singapore（新加坡国立大学）；University of Southern Queensland（南昆士兰大学）
University of Malaga（马拉加大学）；University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）
One of the main open challenges in visual odometry (VO) is the robustness to difficult illumination conditions or high dynamic range (HDR) environments. The main difficulties in these situations come from both the limitations of the sensors and the inability to perform a successful tracking of interest points because of the bold assumptions in VO, such as brightness constancy. We address this problem from a deep learning perspective, for which we first fine-tune a Deep Neural Network (DNN) with the purpose of obtaining enhanced representations of the sequences for VO. Then, we demonstrate how …
Current monocular-based 6D object pose estimation methods generally achieve less competitive results than RGBD-based methods, mostly due to the lack of 3D information. To make up this gap, this paper proposes a 3D geometric volume based pose estimation method with a short baseline two-view setting. By constructing a geometric volume in the 3D space, we combine the features from two adjacent images to the same 3D space. Then a network is trained to learn the distribution of the position of object keypoints in the volume, and a robust soft RANSAC solver is deployed to solve the pose in closed fo…
While many visual ego-motion algorithm variants have been proposed in the past decade, learning based ego-motion estimation methods have seen an increasing attention because of its desirable properties of robustness to image noise and camera calibration independence. In this work, we propose a data-driven approach of fully trainable visual ego-motion estimation for a monocular camera. We use an end-to-end learning approach in allowing the model to map directly from input image pairs to an estimate of ego-motion (parameterized as 6-DoF transformation matrices). We introduce a novel two-module L…
Nagoya University（名古屋大学）；Aichi Institute of Technology（爱知工业大学）；Nagoya University Hospital（名古屋大学医院）；Nagoya University Graduate School of Medicine（名古屋大学医学研究生院）；Technical University of Munich（慕尼黑工业大学）；National Institute of Informatics（国立信息学研究所）
This paper presents a colonoscope tracking method utilizing a colon shape estimation method. CT colonography is used as a less-invasive colon diagnosis method. If colonic polyps or early-stage cancers are found, they are removed in a colonoscopic examination. In the colonoscopic examination, understanding where the colonoscope running in the colon is difficult. A colonoscope navigation system is necessary to reduce overlooking of polyps. We propose a colonoscope tracking method for navigation systems. Previous colonoscope tracking methods caused large tracking errors because they do not consid…
The paper describes an open source computer vision-based hardware structure and software algorithm, which analyzes layer-wise the 3-D printing processes, tracks printing errors, and generates appropriate printer actions to improve reliability. This approach is built upon multiple-stage monocular image examination, which allows monitoring both the external shape of the printed object and internal structure of its layers. Starting with the side-view height validation, the developed program analyzes the virtual top view for outer shell contour correspondence using the multi-template matching and …
