Reconstructing accurate 3D surface models of sinus anatomy directly from an endoscopic video is a promising avenue for cross-sectional and longitudinal analysis to better understand the relationship between sinus anatomy and surgical outcomes. We present a patient-specific, learning-based method for 3D reconstruction of sinus surface anatomy directly and only from endoscopic videos. We demonstrate the effectiveness and accuracy of our method on in and ex vivo data where we compare to sparse reconstructions from Structure from Motion, dense reconstruction from COLMAP, and ground truth anatomy from CT. Our textured reconstructions are watertight and enable measurement of clinically relevant parameters in good agreement with CT. The source code is available at https://github.com/lppllppl920/DenseReconstruction-Pytorch.
The Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
We present a self-supervised learning-based pipeline for dense 3D reconstruction from full-length monocular endoscopic videos without a priori modeling of anatomy or shading. Our method only relies on unlabeled monocular endoscopic videos and conventional multi-view stereo algorithms, and requires neither manual interaction nor patient CT in both training and application phases. In a cross-patient study using CT scans as groundtruth, we show that our method is able to produce photo-realistic dense 3D reconstructions with submillimeter mean residual errors from endoscopic videos from unseen pat…
ExtraGS: Enhancing Endoscopic View Extrapolation via Diffusion-Guided 3D Gaussian Splatting
The School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, and the Shanghai Key Laboratory of Navigation and Location-Based Services（上海交通大学自动化与智能感知学院以及上海市导航与位置服务重点实验室）；Global College, Shanghai Jiao Tong University（上海交通大学密西根学院）；Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属第九人民医院）
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), UCL（伦敦大学学院威康/EPSRC介入与外科科学中心）；National Center for Tumor Diseases (NCT) Dresden（德累斯顿国家肿瘤疾病中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
In computer vision, reference datasets have been highly successful in promoting algorithmic development in stereo reconstruction. Surgical scenes gives rise to specific problems, including the lack of clear corner features, highly specular surfaces and the presence of blood and smoke. Publicly available datasets have been produced using CT and either phantom images or biological tissue samples covering a relatively small region of the endoscope field-of-view. We present a stereo-endoscopic reconstruction validation dataset based on CT (SERV-CT). Two {\it ex vivo} small porcine full torso cadav…
ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting
Jagiellonian University, Kraków, Poland（杰洛尼摩大学）；Warsaw University of Technology, Poland（华沙技术大学）；Sano Centre for Computational Medicine, Kraków, Poland（Sano计算医学中心）；University of Verona, Italy（威尼斯大学）；IDEAS Research Institute, Warsaw, Poland（IDEAS研究机构）
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipel…
EndoLive: Real-Time Style Transfer for Endoscopic Endonasal Skull Base Surgical Video
University of Pittsburgh（匹兹堡大学）；University of Pittsburgh School of Computing and Information（匹兹堡大学计算与信息学院）；School of Computing and Information（计算与信息学院）
F2PASeg: Feature Fusion for Pituitary Anatomy Segmentation in Endoscopic Surgery
1Centre for Artificial Intelligence；Robotics, Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences 2The University of Hong Kong 3Peking Union Medical College Hospital, Chinese Academy of Medical Science
Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
High-quality 3D reconstructions from endoscopy video play an important role in many clinical applications, including surgical navigation where they enable direct video-CT registration. While many methods exist for general multi-view 3D reconstruction, these methods often fail to deliver satisfactory performance on endoscopic video. Part of the reason is that local descriptors that establish pair-wise point correspondences, and thus drive reconstruction, struggle when confronted with the texture-scarce surface of anatomy. Learning-based dense descriptors usually have larger receptive fields ena…
The Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
Functional endoscopic sinus surgery (FESS) is a surgical procedure used to treat acute cases of sinusitis and other sinus diseases. FESS is fast becoming the preferred choice of treatment due to its minimally invasive nature. However, due to the limited field of view of the endoscope, surgeons rely on navigation systems to guide them within the nasal cavity. State of the art navigation systems report registration accuracy of over 1mm, which is large compared to the size of the nasal airways. We present an anatomically constrained video-CT registration algorithm that incorporates multiple video…
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）；Soochow University（苏州大学）
Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environmen…
