Depth estimation from monocular endoscopic images presents significant challenges due to the complexity of endoscopic surgery, such as irregular shapes of human soft tissues, as well as variations in lighting conditions. Existing methods primarily estimate the depth information from RGB images directly, and often surffer the limited interpretability and accuracy. Given that RGB and depth images are two views of the same endoscopic surgery scene, in this paper, we introduce a novel concept referred as ``meta feature embedding (MetaFE)", in which the physical entities (e.g., tissues and surgical instruments) of endoscopic surgery are represented using the shared features that can be alternatively decoded into RGB or depth image. With this concept, we propose a two-stage self-supervised learning paradigm for the monocular endoscopic depth estimation. In the first stage, we propose a temporal representation learner using diffusion models, which are aligned with the spatial information through the cross normalization to construct the MetaFE. In the second stage, self-supervised monocular depth estimation with the brightness calibration is applied to decode the meta features into the depth image. Extensive evaluation on diverse endoscopic datasets demonstrates that our approach outperforms the state-of-the-art method in depth estimation, achieving superior accuracy and generalization. The source code will be publicly available.
The Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
We present a self-supervised learning-based pipeline for dense 3D reconstruction from full-length monocular endoscopic videos without a priori modeling of anatomy or shading. Our method only relies on unlabeled monocular endoscopic videos and conventional multi-view stereo algorithms, and requires neither manual interaction nor patient CT in both training and application phases. In a cross-patient study using CT scans as groundtruth, we show that our method is able to produce photo-realistic dense 3D reconstructions with submillimeter mean residual errors from endoscopic videos from unseen pat…
Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
The Chinese University of Hong Kong (CUHK)（香港中文大学）；University College London（伦敦大学学院）；Shun Hing Institute of Advanced Engineering, CUHK（香港中文大学信兴高等工程研究所）；Shenzhen Research Institute, CUHK（香港中文大学深圳研究院）
EndoGMDE: Generalizable Monocular Depth Estimation with Mixture of Low-Rank Experts for Diverse Endoscopic Scenes
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；College of Biomedical Engineering, Fudan University（复旦大学生物医学工程学院）；Shanghai Key laboratory of Medical Image Computing and Computer Assisted Intervention（上海医学影像计算与计算机辅助手术重点实验室）；ENT Institute and Department of Otolaryngology, Eye & ENT Hospital of Fudan University（复旦大学眼耳鼻喉科医院耳鼻喉科及耳鼻喉科研究所）
MonoPCC: Photometric-invariant Cycle Constraint for Monocular Depth Estimation of Endoscopic Images
Huazhong University of Science and Technology（华中科技大学）；Wuhan United Imaging Healthcare Surgical Technology Co., Ltd.（武汉联影外科技术有限公司）；Tongji Hospital（同济医院）；Tongji Medical College（同济医学院）；Britton Chance Center for Biomedical Photonics（布里顿·钱斯生物光子学中心）；Wuhan National Laboratory for Optoelectronics（武汉光电国家实验室）
Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Video
The Chinese University of Hong Kong（香港中文大学）；Hong Kong Center for Logistics Robotics（香港物流机器人中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiao Tong University（上海交通大学）
Institute of Biomedical Engineering, Bogazici University（波加兹奇大学生物医学工程研究所）；Computer Science Department, University of Oxford（牛津大学计算机科学系）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布莱根妇女医院）；Cancer Data Science, Dana Farber Cancer Institute（丹娜-法伯癌症研究所癌症数据科学部）；Cancer Program, Broad Institute of Harvard and MIT（哈佛-麻省理工学院布罗德研究所癌症项目）；Institute for Systems and Robotics, University of Coimbra（科英布拉大学系统与机器人研究所）；Faculty of Medicine, Clinical Academic Center of Coimbra, University of Coimbra（科英布拉大学科英布拉临床学术中心医学院）；Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）
Deep learning techniques hold promise to develop dense topography reconstruction and pose estimation methods for endoscopic videos. However, currently available datasets do not support effective quantitative benchmarking. In this paper, we introduce a comprehensive endoscopic SLAM dataset consisting of 3D point cloud data for six porcine organs, capsule and standard endoscopy recordings as well as synthetically generated data. A Panda robotic arm, two commercially available capsule endoscopes, two conventional endoscopes with different camera properties, and two high precision 3D scanners were…
Nagoya University（名古屋大学）；Kyoto Second Red Cross Hospital（京都第二红十字医院）；Sapporo-Minami-Sanjo Hospital（札幌南三条医院）；Sapporo-Kosei General Hospital（札幌厚生综合医院）；Keiwakai Nishioka Hospital（惠和会西冈医院）；National Institute of Informatics（国立信息学研究所）
We propose a depth estimation method from a single-shot monocular endoscopic image using Lambertian surface translation by domain adaptation and depth estimation using multi-scale edge loss. We employ a two-step estimation process including Lambertian surface translation from unpaired data and depth estimation. The texture and specular reflection on the surface of an organ reduce the accuracy of depth estimations. We apply Lambertian surface translation to an endoscopic image to remove these texture and reflections. Then, we estimate the depth by using a fully convolutional network (FCN). Duri…
EndoSfM3D: Learning to 3D Reconstruct Any Endoscopic Surgery Scene using Self-supervised Foundation Model
UCL Hawkes Institute（UCL哈维斯研究所）；Department of Medical Physics（医学物理系）；Biomedical Engineering, University College London, UK（生物医学工程，伦敦大学学院，英国）；Division of Informatics, Imaging and Data Sciences, The University of Manchester, Manchester, UK（信息学、成像与数据科学部门，曼彻斯特大学，曼彻斯特，英国）
Indian Institute of Technology Guwahati（印度理工学院古瓦哈蒂分校）；Chubu University（中部大学）
A deep learning-based monocular depth estimation (MDE) technique is proposed for selection of most informative frames (key frames) of an endoscopic video. In most of the cases, ground truth depth maps of polyps are not readily available and that is why the transfer learning approach is adopted in our method. An endoscopic modalities generally capture thousands of frames. In this scenario, it is quite important to discard low-quality and clinically irrelevant frames of an endoscopic video while the most informative frames should be retained for clinical diagnosis. In this view, a key-frame sele…
The Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires sequential data from monocular endoscopic videos and a multi-view stereo reconstruction method, e.g. structure from motion, that supervises learning in a sparse but accurate manner. Consequently, our method requires neither manual interaction, such as scaling or labeling, nor patient CT in the training and application phases. We demonstrate the performance of our method on sinus endosco…
4D Monocular Surgical Reconstruction under Arbitrary Camera Motions
Department of Mechanical and Automation Engineering（机械与自动化工程系）；T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong（T Stone机器人研究院，香港中文大学，香港）；Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China（沈阳自动化研究所，中国科学院，沈阳，中国）；State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China（机器人与智能系统国家重点实验室，中国，沈阳）；School of Integrated Circuits, Shanghai Jiao Tong University, Shanghai, China（集成电路学院，上海交通大学，中国，上海）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China（自动化与智能感知学院，上海交通大学，中国，上海）；Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China（系统控制与信息处理重点实验室，中国教育部，中国，上海）
Recently, self-supervised learning technology has been applied to calculate depth and ego-motion from monocular videos, achieving remarkable performance in autonomous driving scenarios. One widely adopted assumption of depth and ego-motion self-supervised learning is that the image brightness remains constant within nearby frames. Unfortunately, the endoscopic scene does not meet this assumption because there are severe brightness fluctuations induced by illumination variations, non-Lambertian reflections and interreflections during data collection, and these brightness fluctuations inevitably…
DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/英国工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；Politecnico di Milano（米兰理工大学）；Centre for Medical Image Computing（医学影像计算中心）
Medical endoscopy remains a challenging application for simultaneous localization and mapping (SLAM) due to the sparsity of image features and size constraints that prevent direct depth-sensing. We present a SLAM approach that incorporates depth predictions made by an adversarially-trained convolutional neural network (CNN) applied to monocular endoscopy images. The depth network is trained with synthetic images of a simple colon model, and then fine-tuned with domain-randomized, photorealistic images rendered from computed tomography measurements of human colons. Each image is paired with an …
Johns Hopkins University（约翰斯·霍普金斯大学）；Philips Research（飞利浦研究院）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）；Facebook（脸书）
We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires monocular endoscopic videos and a multi-view stereo method, e.g., structure from motion, to supervise learning in a sparse manner. Consequently, our method requires neither manual labeling nor patient computed tomography (CT) scan in the training and application phases. In a cross-patient experiment using CT scans as groundtruth, the proposed method achieved submillimeter mean residual …
University Hospital Heidelberg（海德堡大学医院）；Otto-von-Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）
Minimally Invasive Surgery (MIS) techniques have gained rapid popularity among surgeons since they offer significant clinical benefits including reduced recovery time and diminished post-operative adverse effects. However, conventional endoscopic systems output monocular video which compromises depth perception, spatial orientation and field of view. Suturing is one of the most complex tasks performed under these circumstances. Key components of this tasks are the interplay between needle holder and the surgical needle. Reliable 3D localization of needle and instruments in real time could be u…
A geometry-aware deep network for depth estimation in monocular endoscopy
Northeastern University（东北大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；Beihang University（北京航空航天大学）；General Hospital of Northern Theater Command（北部战区总医院）
NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong（机械与自动化工程系和T Stone机器人研究所，香港中文大学）；School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University（自动化与智能感知学院、航空系统集成与航空系统综合国家重点实验室、上海导航与定位服务重点实验室，上海交通大学）
Endoscopic Depth Estimation Based on Deep Learning: A Survey
Beijing Information Science and Technology University（北京信息科技大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（南方科技大学可信自主系统研究院）；The Third Hospital of Xiamen（厦门市第三医院）；Fujian Institute of Geriatrics（福建省老年医学研究所）；Fujian Medical University Union Hospital（福建医科大学附属协和医院）；Beijing Tiromu Medical Technology Co., Ltd.（北京提罗木医疗科技有限公司）；National University of Singapore（新加坡国立大学）；University of Southern Queensland（南昆士兰大学）
Bournemouth University（伯恩茅斯大学）；University of Chester（切斯特大学）；University of Bradford（布拉德福德大学）
One of the major challenges in Minimally Invasive Surgery (MIS) such as laparoscopy is the lack of depth perception. In recent years, laparoscopic scene tracking and surface reconstruction has been a focus of investigation to provide rich additional information to aid the surgical process and compensate for the depth perception issue. However, robust 3D surface reconstruction and augmented reality with depth perception on the reconstructed scene are yet to be reported. This paper presents our work in this area. First, we adopt a state-of-the-art visual simultaneous localization and mapping (SL…
EndoMUST: Monocular Depth Estimation for Robotic Endoscopy via End-to-end Multi-step Self-supervised Training
Academy for Engineering & Technology, Fudan University（工程与技术学院，复旦大学）；Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Fudan University（上海医学图像计算与计算机辅助干预重点实验室，复旦大学）
Surgical robots are used to perform minimally invasive surgery and alleviate much of the burden imposed on surgeons. Our group has developed a surgical robot to aid in the removal of tumors at the base of the skull via access through the nostrils. To avoid injuring the patients, a collision-avoidance algorithm that depends on having an accurate model for the poses of the instruments' shafts is used. Given that the model's parameters can change over time owing to interactions between instruments and other disturbances, the online estimation of the poses of the instrument's shaft is essential. I…
Bridging the Ex-Vivo to In-Vivo Gap: Synthetic Priors for Monocular Depth Estimation in Specular Surgical Environments
RoCAL, Rochester Institute of Technology（罗切斯特理工学院RoCAL实验室）；Department of Otolaryngology Head and Neck Surgery, University of Rochester Medical Center（罗切斯特大学医学中心耳鼻喉科与头颈外科部门）；University of Washington, Department of Otolaryngology–Head and Neck Surgery（华盛顿大学耳鼻喉科与头颈外科部门）
