SERV-CT: A disparity dataset from CT for validation of endoscopic 3D reconstruction
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), UCL（伦敦大学学院威康/EPSRC介入与外科科学中心）；National Center for Tumor Diseases (NCT) Dresden（德累斯顿国家肿瘤疾病中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
P. J. "Eddie'' Edwards, Dimitris Psychogyios, Stefanie Speidel, Lena Maier-Hein, Danail Stoyanov
备注：
Submitted to Medical Image Analysis. 14 Figures, 17 pages
In computer vision, reference datasets have been highly successful in promoting algorithmic development in stereo reconstruction. Surgical scenes gives rise to specific problems, including the lack of clear corner features, highly specular surfaces and the presence of blood and smoke. Publicly available datasets have been produced using CT and either phantom images or biological tissue samples covering a relatively small region of the endoscope field-of-view. We present a stereo-endoscopic reconstruction validation dataset based on CT (SERV-CT). Two {\it ex vivo} small porcine full torso cadavers were placed within the view of the endoscope with both the endoscope and target anatomy visible in the CT scan. Orientation of the endoscope was manually aligned to the stereoscopic view. Reference disparities and occlusions were calculated for 8 stereo pairs from each sample. For the second sample an RGB surface was acquired to aid alignment of smooth, featureless surfaces. Repeated manual alignments showed an RMS disparity accuracy of ~2 pixels and a depth accuracy of ~2mm. The reference dataset includes endoscope image pairs with corresponding calibration, disparities, depths and occlusions covering the majority of the endoscopic image and a range of tissue types. Smooth specular surfaces and images with significant variation of depth are included. We assessed the performance of various stereo algorithms from online available repositories. There is a significant variation between algorithms, highlighting some of the challenges of surgical endoscopic images. The SERV-CT dataset provides an easy to use stereoscopic validation for surgical applications with smooth reference disparities and depths with coverage over the majority of the endoscopic images. This complements existing resources well and we hope will aid the development of surgical endoscopic anatomical reconstruction algorithms.
ETH Zurich（苏黎世联邦理工学院）；San Raffaele Hospital（圣拉斐尔医院）；University of Verona（维罗纳大学）；Oxford Brookes University（牛津布鲁克斯大学）
In this work, we take aim towards increasing the effectiveness of surgical assistant robots. We intended to make assistant robots safer by making them aware about the actions of surgeon, so it can take appropriate assisting actions. In other words, we aim to solve the problem of surgeon action detection in endoscopic videos. To this, we introduce a challenging dataset for surgeon action detection in real-world endoscopic videos. Action classes are picked based on the feedback of surgeons and annotated by medical professional. Given a video frame, we draw bounding box around surgical tool which…
Institute of Image Processing（图像处理研究所）；Pattern Recognition Institute of Medical Robotics（医学机器人模式识别研究所）；Shanghai Jiao Tong University（上海交通大学）
With the popularity of stereo cameras in computer assisted surgery techniques, a second viewpoint would provide additional information in surgery. However, how to effectively access and use stereo information for the super-resolution (SR) purpose is often a challenge. In this paper, we propose a disparity-constrained stereo super-resolution network (DCSSRnet) to simultaneously compute a super-resolved image in a stereo image pair. In particular, we incorporate a disparity-based constraint mechanism into the generation of SR images in a deep neural network framework with an additional atrous pa…
EndoGMDE: Generalizable Monocular Depth Estimation with Mixture of Low-Rank Experts for Diverse Endoscopic Scenes
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；College of Biomedical Engineering, Fudan University（复旦大学生物医学工程学院）；Shanghai Key laboratory of Medical Image Computing and Computer Assisted Intervention（上海医学影像计算与计算机辅助手术重点实验室）；ENT Institute and Department of Otolaryngology, Eye & ENT Hospital of Fudan University（复旦大学眼耳鼻喉科医院耳鼻喉科及耳鼻喉科研究所）
Robust Real-Time Endoscopic Stereo Matching under Fuzzy Tissue Boundaries
The School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China（生物医学工程学院，上海交通大学）；Innovation Center for Electronic Design Automation Technology, Hangzhou Dianzi University, Hangzhou 310018, China（电子设计自动化技术创新中心，杭州电子科技大学）
Bridging the Ex-Vivo to In-Vivo Gap: Synthetic Priors for Monocular Depth Estimation in Specular Surgical Environments
RoCAL, Rochester Institute of Technology（罗切斯特理工学院RoCAL实验室）；Department of Otolaryngology Head and Neck Surgery, University of Rochester Medical Center（罗切斯特大学医学中心耳鼻喉科与头颈外科部门）；University of Washington, Department of Otolaryngology–Head and Neck Surgery（华盛顿大学耳鼻喉科与头颈外科部门）
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Department of Mechanical Engineering, Carnegie Mellon University（卡内基梅隆大学机械工程系）
This work presents dense stereo reconstruction using high-resolution images for infrastructure inspections. The state-of-the-art stereo reconstruction methods, both learning and non-learning ones, consume too much computational resource on high-resolution data. Recent learning-based methods achieve top ranks on most benchmarks. However, they suffer from the generalization issue due to lack of task-specific training data. We propose to use a less resource demanding non-learning method, guided by a learning-based model, to handle high-resolution images and achieve accurate stereo reconstruction.…
Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from…
University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）
Event-based cameras are biologically inspired sensors that output asynchronous pixel-wise brightness changes in the scene called events. They have a high dynamic range and temporal resolution of a microsecond, opposed to standard cameras that output frames at fixed frame rates and suffer from motion blur. Forming stereo pairs of such cameras can open novel application possibilities, since for each event depth can be readily estimated; however, to fully exploit asynchronous nature of the sensor and avoid fixed time interval event accumulation, stereo event lifetime estimation should be employed…
Oxford Brookes University（牛津布鲁克斯大学）；ETH Zurich（苏黎世联邦理工学院）；San Raffaele Hospital（圣拉斐尔医院）；Harbin Institute of Technology（哈尔滨工业大学）；Osaka University（大阪大学）；The George Washington University（乔治华盛顿大学）；University of Verona（维罗纳大学）
For an autonomous robotic system, monitoring surgeon actions and assisting the main surgeon during a procedure can be very challenging. The challenges come from the peculiar structure of the surgical scene, the greater similarity in appearance of actions performed via tools in a cavity compared to, say, human actions in unconstrained environments, as well as from the motion of the endoscopic camera. This paper presents ESAD, the first large-scale dataset designed to tackle the problem of surgeon action detection in endoscopic minimally invasive surgery. ESAD aims at contributing to increase th…
EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd.（华为技术有限公司中央研究院理论实验室2012实验室）；Chair of Computer Aided Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助程序教席）；Department of Gastroenterology, Qilu Hospital of Shandong University（山东大学齐鲁医院消化内科）
The condition assessment of road surfaces is essential to ensure their serviceability while still providing maximum road traffic safety. This paper presents a robust stereo vision system embedded in an unmanned aerial vehicle (UAV). The perspective view of the target image is first transformed into the reference view, and this not only improves the disparity accuracy, but also reduces the algorithm's computational complexity. The cost volumes generated from stereo matching are then filtered using a bilateral filter. The latter has been proved to be a feasible solution for the functional minimi…
We introduce the task of stereo video reconstruction or, equivalently, 2D-to-3D video conversion for minimally invasive surgical video. We design and implement a series of end-to-end U-Net-based solutions for this task by varying the input (single frame vs. multiple consecutive frames), loss function (MSE, MAE, or perceptual losses), and network architecture. We evaluate these solutions by surveying ten experts - surgeons who routinely perform endoscopic surgery. We run two separate reader studies: one evaluating individual frames and the other evaluating fully reconstructed 3D video played on…
A Client-server Deep Federated Learning for Cross-domain Surgical Image Segmentation
NepAl Applied Mathematics and Informatics Institute (NAAMII)（尼泊尔应用数学与信息学研究所（NAAMII））；University of Leeds（利兹大学）；University of Liverpool（利物浦大学）；University College London（伦敦大学学院）；University of Aberdeen（阿伯丁大学）
University of Tübingen（图宾根大学）；Mercedes-Benz AG（梅赛德斯-奔驰股份公司）
The sparse coding algorithm has served as a model for early processing in mammalian vision. It has been assumed that the brain uses sparse coding to exploit statistical properties of the sensory stream. We hypothesize that sparse coding discovers patterns from the data set, which can be used to estimate a set of stimulus parameters by simple readout. In this study, we chose a model of stereo vision to test our hypothesis. We used the Locally Competitive Algorithm (LCA), followed by a naïve Bayes classifier, to infer stereo disparity. From the results we report three observations. First, dispar…
Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)
Regensburg Medical Image Computing (ReMIC), OTH Regensburg（雷根斯堡医学影像计算中心（ReMIC））；Regensburg Center of Biomedical Engineering (RCBE), OTH Regensburg（雷根斯堡生物医学工程中心（RCBE））；Regensburg University（雷根斯堡大学）；Regensburg Center of Health Sciences（雷根斯堡健康科学中心）；Technical University of Munich (TUM)（慕尼黑技术大学）；TUM University Hospital（慕尼黑技术大学医院）；Biomedical Image Analysis Group, Department of Computing, Imperial College London（生物医学影像分析组，计算部门，伦敦帝国学院）；Research Group MITI, TUM University Hospital, School of Medicine（MITI研究组，慕尼黑技术大学医院，医学学院）；Department of Surgery, TUM University Hospital, School of Medicine（外科部门，慕尼黑技术大学医院，医学学院）
Deep learning-based quality metrics have recently given significant improvement in Image Quality Assessment (IQA). In the field of stereoscopic vision, information is evenly distributed with slight disparity to the left and right eyes. However, due to asymmetric distortion, the objective quality ratings for the left and right images would differ, necessitating the learning of unique quality indicators for each view. Unlike existing stereoscopic IQA measures which focus mainly on estimating a global human score, we suggest incorporating left, right, and stereoscopic objective scores to extract …
