EndoSfM3D: Learning to 3D Reconstruct Any Endoscopic Surgery Scene using Self-supervised Foundation Model
UCL Hawkes Institute（UCL哈维斯研究所）；Department of Medical Physics（医学物理系）；Biomedical Engineering, University College London, UK（生物医学工程，伦敦大学学院，英国）；Division of Informatics, Imaging and Data Sciences, The University of Manchester, Manchester, UK（信息学、成像与数据科学部门，曼彻斯特大学，曼彻斯特，英国）
Changhao Zhang, Matthew J. Clarkson, Mobarak I. Hoque
3D reconstruction of endoscopic surgery scenes plays a vital role in enhancing scene perception, enabling AR visualization, and supporting context-aware decision-making in image-guided surgery. A critical yet challenging step in this process is the accurate estimation of the endoscope's intrinsic parameters. In real surgical settings, intrinsic calibration is hindered by sterility constraints and the use of specialized endoscopes with continuous zoom and telescope rotation. Most existing methods for endoscopic 3D reconstruction do not estimate intrinsic parameters, limiting their effectiveness for accurate and reliable reconstruction. In this paper, we integrate intrinsic parameter estimation into a self-supervised monocular depth estimation framework by adapting the Depth Anything V2 (DA2) model for joint depth, pose, and intrinsics prediction. We introduce an attention-based pose network and a Weight-Decomposed Low-Rank Adaptation (DoRA) strategy for efficient fine-tuning of DA2. Our method is validated on the SCARED and C3VD public datasets, demonstrating superior performance compared to recent state-of-the-art approaches in self-supervised monocular depth estimation and 3D reconstruction. Code and model weights can be found in project repository: https://github.com/MOYF-beta/EndoSfM3D.
DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/英国工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；Politecnico di Milano（米兰理工大学）；Centre for Medical Image Computing（医学影像计算中心）
Real-Time Spatio-Temporal Reconstruction of Dynamic Endoscopic Scenes with 4D Gaussian Splatting
University of Liverpool, Liverpool, UK（利物浦大学）；Xi'an Jiaotong-Liverpool University, Suzhou, China（西安交通大学利物浦大学）；University of Cambridge, Cambridge, UK（剑桥大学）
EndoMatcher: Generalizable Endoscopic Image Matcher via Multi-Domain Pre-training for Robot-Assisted Surgery
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院呼吸与危重症医学科）；Hong Kong Institute of Science & Innovation（香港创新科技署）；Centre of AI and Robotics, Hong Kong Institute of Science & Innovation（人工智能与机器人中心，香港创新科技署）
Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心）；National University of Singapore（新加坡国立大学）
The Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
We present a self-supervised learning-based pipeline for dense 3D reconstruction from full-length monocular endoscopic videos without a priori modeling of anatomy or shading. Our method only relies on unlabeled monocular endoscopic videos and conventional multi-view stereo algorithms, and requires neither manual interaction nor patient CT in both training and application phases. In a cross-patient study using CT scans as groundtruth, we show that our method is able to produce photo-realistic dense 3D reconstructions with submillimeter mean residual errors from endoscopic videos from unseen pat…
The Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
Reconstructing accurate 3D surface models of sinus anatomy directly from an endoscopic video is a promising avenue for cross-sectional and longitudinal analysis to better understand the relationship between sinus anatomy and surgical outcomes. We present a patient-specific, learning-based method for 3D reconstruction of sinus surface anatomy directly and only from endoscopic videos. We demonstrate the effectiveness and accuracy of our method on in and ex vivo data where we compare to sparse reconstructions from Structure from Motion, dense reconstruction from COLMAP, and ground truth anatomy f…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）；Middle East Technical University（中东技术大学）；Sabanci University（萨班哲大学）；University of Coimbra（科英布拉大学）
Since the development of capsule endoscopcy technology, substantial progress were made in converting passive capsule endoscopes to robotic active capsule endoscopes which can be controlled by the doctor. However, robotic capsule endoscopy still has some challenges. In particular, the use of such devices to generate a precise and globally consistent three-dimensional (3D) map of the entire inner organ remains an unsolved problem. Such global 3D maps of inner organs would help doctors to detect the location and size of diseased areas more accurately, precisely, and intuitively, thus permitting m…
ExtraGS: Enhancing Endoscopic View Extrapolation via Diffusion-Guided 3D Gaussian Splatting
The School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, and the Shanghai Key Laboratory of Navigation and Location-Based Services（上海交通大学自动化与智能感知学院以及上海市导航与位置服务重点实验室）；Global College, Shanghai Jiao Tong University（上海交通大学密西根学院）；Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属第九人民医院）
SAGS: Self-Adaptive Alias-Free Gaussian Splatting for Dynamic Surgical Endoscopic Reconstruction
University of Technology Sydney, Australia（悉尼技术大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China（深圳先进技术研究所）；Shenyang Institute of Automation, Chinese Academy of Sciences, China（沈阳自动化研究所）
Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
The Chinese University of Hong Kong (CUHK)（香港中文大学）；University College London（伦敦大学学院）；Shun Hing Institute of Advanced Engineering, CUHK（香港中文大学信兴高等工程研究所）；Shenzhen Research Institute, CUHK（香港中文大学深圳研究院）
Intuitive Inc.（直觉外科公司）；Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (HHI)（弗劳恩霍夫海因里希·赫兹通信研究所）；University of Western Ontario（西安大略大学）；Digital Surgery Ltd.（数字外科有限公司）；Norwegian University of Science and Technology（挪威科技大学）；Rediminds Inc.（Rediminds公司）；Harbin Institute of Technology（哈尔滨工业大学）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Heidelberg University（海德堡大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) UCL（伦敦大学学院惠康/EPSRC介入与外科科学中心）；University of Bern（伯尔尼大学）
The stereo correspondence and reconstruction of endoscopic data sub-challenge was organized during the Endovis challenge at MICCAI 2019 in Shenzhen, China. The task was to perform dense depth estimation using 7 training datasets and 2 test sets of structured light data captured using porcine cadavers. These were provided by a team at Intuitive Surgical. 10 teams participated in the challenge day. This paper contains 3 additional methods which were submitted after the challenge finished as well as a supplemental section from these teams on issues they found with the dataset.
In the gastrointestinal (GI) tract endoscopy field, ingestible wireless capsule endoscopy is emerging as a novel, minimally invasive diagnostic technology for inspection of the GI tract and diagnosis of a wide range of diseases and pathologies. Since the development of this technology, medical device companies and many research groups have made substantial progress in converting passive capsule endoscopes to robotic active capsule endoscopes with most of the functionality of current active flexible endoscopes. However, robotic capsule endoscopy still has some challenges. In particular, the use…
Reconstruction of the soft tissues in robotic surgery from endoscopic stereo videos is important for many applications such as intra-operative navigation and image-guided robotic surgery automation. Previous works on this task mainly rely on SLAM-based approaches, which struggle to handle complex surgical scenes. Inspired by recent progress in neural rendering, we present a novel framework for deformable tissue reconstruction from binocular captures in robotic surgery under the single-viewpoint setting. Our framework adopts dynamic neural radiance fields to represent deformable surgical scenes…
Asan Medical Center（峨山医疗中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Chungnam National University School of Medicine（忠南大学医学院）
Some endoscopic surgeries require a surgeon to hold the endoscope with one hand and the surgical instruments with the other hand to perform the actual surgery with correct vision. Recent technical advances in deep learning as well as in robotics can introduce robotics to these endoscopic surgeries. This can have numerous advantages by freeing one hand of the surgeon, which will allow the surgeon to use both hands and to use more intricate and sophisticated techniques. Recently, deep learning with convolutional neural network achieves state-of-the-art results in computer vision. Therefore, the …
Gaussian Process Prior Variational Autoencoder for Endoscopic Videos
Department of Electromechanics, InViLab, University of Antwerp（安特卫普大学机电工程系InViLab实验室）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Department of Electrical Engineering, Eindhoven University of Technology（埃因霍温理工大学电气工程系）
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), UCL（伦敦大学学院威康/EPSRC介入与外科科学中心）；National Center for Tumor Diseases (NCT) Dresden（德累斯顿国家肿瘤疾病中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
In computer vision, reference datasets have been highly successful in promoting algorithmic development in stereo reconstruction. Surgical scenes gives rise to specific problems, including the lack of clear corner features, highly specular surfaces and the presence of blood and smoke. Publicly available datasets have been produced using CT and either phantom images or biological tissue samples covering a relatively small region of the endoscope field-of-view. We present a stereo-endoscopic reconstruction validation dataset based on CT (SERV-CT). Two {\it ex vivo} small porcine full torso cadav…
Endo-4DGX: Robust Endoscopic Scene Reconstruction and Illumination Correction with Gaussian Splatting
Department of Electronic Engineering, The Chinese University of Hong Kong (CUHK), Hong Kong SAR, China（香港中文大学电子工程系）；Shun Hing Institute of Advanced Engineering, CUHK, Hong Kong SAR, China（CUHK先进工程研究所）；City University of Hong Kong, Hong Kong, China（香港城市大学）；The University of Sydney, Sydney, NSW, Australia（悉尼大学）；Centre for Artificial Intelligence（人工智能中心）
Indian Institute of Technology Guwahati（印度理工学院古瓦哈蒂分校）；Chubu University（中部大学）
A deep learning-based monocular depth estimation (MDE) technique is proposed for selection of most informative frames (key frames) of an endoscopic video. In most of the cases, ground truth depth maps of polyps are not readily available and that is why the transfer learning approach is adopted in our method. An endoscopic modalities generally capture thousands of frames. In this scenario, it is quite important to discard low-quality and clinically irrelevant frames of an endoscopic video while the most informative frames should be retained for clinical diagnosis. In this view, a key-frame sele…
EndoWave: 4D Gaussian Splatting with Rational Wavelet for Endoscopic Reconstruction
Xi'an Jiaotong-Liverpool University（西安交通大学利物浦大学）；University of Liverpool（利物浦大学）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Zhejiang University of Technology（浙江工业大学）
Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping
The Chinese University of Hong Kong (CUHK)（香港中文大学）；CUHK Shenzhen Research Institute（香港中文大学深圳研究院）；Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）；Qilu Hospital of Shandong University（山东大学齐鲁医院）
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipel…
ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting
Jagiellonian University, Kraków, Poland（杰洛尼摩大学）；Warsaw University of Technology, Poland（华沙技术大学）；Sano Centre for Computational Medicine, Kraków, Poland（Sano计算医学中心）；University of Verona, Italy（威尼斯大学）；IDEAS Research Institute, Warsaw, Poland（IDEAS研究机构）
Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Video
The Chinese University of Hong Kong（香港中文大学）；Hong Kong Center for Logistics Robotics（香港物流机器人中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiao Tong University（上海交通大学）
Bournemouth University（伯恩茅斯大学）；University of Chester（切斯特大学）；University of Bradford（布拉德福德大学）
One of the major challenges in Minimally Invasive Surgery (MIS) such as laparoscopy is the lack of depth perception. In recent years, laparoscopic scene tracking and surface reconstruction has been a focus of investigation to provide rich additional information to aid the surgical process and compensate for the depth perception issue. However, robust 3D surface reconstruction and augmented reality with depth perception on the reconstructed scene are yet to be reported. This paper presents our work in this area. First, we adopt a state-of-the-art visual simultaneous localization and mapping (SL…
