A critical complication after cataract surgery is the dislocation of the lens implant leading to vision deterioration and eye trauma. In order to reduce the risk of this complication, it is vital to discover the risk factors during the surgery. However, studying the relationship between lens dislocation and its suspicious risk factors using numerous videos is a time-extensive procedure. Hence, the surgeons demand an automatic approach to enable a larger-scale and, accordingly, more reliable study. In this paper, we propose a novel framework as the major step towards lens irregularity detection. In particular, we propose (I) an end-to-end recurrent neural network to recognize the lens-implantation phase and (II) a novel semantic segmentation network to segment the lens and pupil after the implantation phase. The phase recognition results reveal the effectiveness of the proposed surgical phase recognition approach. Moreover, the segmentation results confirm the proposed segmentation network's effectiveness compared to state-of-the-art rival approaches.
University College London (UCL)（伦敦大学学院）；King’s College London (KCL)（伦敦国王学院）；Moorfields Eye Hospital（摩尔菲尔兹眼科医院）；Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences（惠康/英国工程与物理科学研究委员会介入与外科科学中心）；School of Biomedical Engineering & Imaging Sciences（生物医学工程与影像科学学院）
Our work proposes neural network design choices that set the state-of-the-art on a challenging public benchmark on cataract surgery, CaDIS. Our methodology achieves strong performance across three semantic segmentation tasks with increasingly granular surgical tool class sets by effectively handling class imbalance, an inherent challenge in any surgical video. We consider and evaluate two conceptually simple data oversampling methods as well as different loss functions. We show significant performance gains across network architectures and tasks especially on the rarest tool classes, thereby p…
Department of Information Technology, Alpen-Adria-Universität Klagenfurt（克恩滕州阿尔卑斯-亚得里亚大学信息技术系）；Department of Ophthalmology, Klinikum Klagenfurt（克拉根福医院眼科）
Semantic segmentation in surgical videos is a prerequisite for a broad range of applications towards improving surgical outcomes and surgical video analysis. However, semantic segmentation in surgical videos involves many challenges. In particular, in cataract surgery, various features of the relevant objects such as blunt edges, color and context variation, reflection, transparency, and motion blur pose a challenge for semantic segmentation. In this paper, we propose a novel convolutional module termed as \textit{ReCal} module, which can calibrate the feature maps by employing region intra-an…
Semantic segmentation in cataract surgery has a wide range of applications contributing to surgical outcome enhancement and clinical risk reduction. However, the varying issues in segmenting the different relevant instances make the designation of a unique network quite challenging. This paper proposes a semantic segmentation network termed as DeepPyram that can achieve superior performance in segmenting relevant objects in cataract surgery videos with varying issues. This superiority mainly originates from three modules: (i) Pyramid View Fusion, which provides a varying-angle global view of t…
Deep-Learning-Assisted Analysis of Cataract Surgery Videos
Alpen-Adria-Universität Klagenfurt（阿尔卑斯-亚得里亚大学克拉根福分校）；University of Applied Sciences Western Switzerland（瑞士西部应用科学大学）；University of Bern（伯尔尼大学）；University of Geneva（日内瓦大学）
Digital Surgery Ltd, Medtronic（美敦力数字外科有限公司）；ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube实验室、法国国家科学研究中心、斯特拉斯堡大学医院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hutom（Hutom公司）；Xiamen University（厦门大学）；University of Electronic Science and Technology of China（电子科技大学）；National University of Singapore（新加坡国立大学）；National Institute of Technology, Raipur（赖布尔国家理工学院）；King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
Surgical scene segmentation is essential for anatomy and instrument localization which can be further used to assess tissue-instrument interactions during a surgical procedure. In 2017, the Challenge on Automatic Tool Annotation for cataRACT Surgery (CATARACTS) released 50 cataract surgery videos accompanied by instrument usage annotations. These annotations included frame-level instrument presence information. In 2020, we released pixel-wise semantic annotations for anatomy and instruments for 4670 images sampled from 25 videos of the CATARACTS training set. The 2020 CATARACTS Semantic Segmen…
University of Bern（伯尔尼大学）；Alpen-Adria-Universität Klagenfurt（克拉根福阿尔卑斯-亚德里亚大学）
Semantic segmentation in cataract surgery has a wide range of applications contributing to surgical outcome enhancement and clinical risk reduction. However, the varying issues in segmenting the different relevant structures in these surgeries make the designation of a unique network quite challenging. This paper proposes a semantic segmentation network, termed DeepPyramid, that can deal with these challenges using three novelties: (1) a Pyramid View Fusion module which provides a varying-angle global view of the surrounding region centering at each pixel position in the input convolutional fe…
CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation
Thomas Jefferson High School for Science and Technology（托马斯·杰弗逊高中科学与技术学校）；Mass Eye and Ear, Harvard Medical School（麻省眼耳研究所，哈佛医学院）；Harvard Ophthalmology AI Lab, Schepens Eye Research Institute of Mass Eye and Ear, Harvard Medical School（哈佛眼科学人工智能实验室，麻省眼耳研究所的谢普恩斯眼研究学院，哈佛医学院）
Digital Surgery LTD（数字手术有限公司）；Inserm（法国国家健康与医学研究院）
Video feedback provides a wealth of information about surgical procedures and is the main sensory cue for surgeons. Scene understanding is crucial to computer assisted interventions (CAI) and to post-operative analysis of the surgical procedure. A fundamental building block of such capabilities is the identification and localization of surgical instruments and anatomical structures through semantic segmentation. Deep learning has advanced semantic segmentation techniques in the recent years but is inherently reliant on the availability of labelled datasets for model training. This paper introd…
University of Bern（伯尔尼大学）；Inselspital, University Hospital, University of Bern（伯尔尼大学附属医院（因塞尔医院））；ARTORG Center, University of Bern（伯尔尼大学ARTORG中心）
Cataract surgery is a sight saving surgery that is performed over 10 million times each year around the world. With such a large demand, the ability to organize surgical wards and operating rooms efficiently is critical to delivery this therapy in routine clinical care. In this context, estimating the remaining surgical duration (RSD) during procedures is one way to help streamline patient throughput and workflows. To this end, we propose CataNet, a method for cataract surgeries that predicts in real time the RSD jointly with two influential elements: the surgeon's experience, and the current …
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Semantic segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, accurate segmentation of cataract surgical instruments is still a challenge due to specular reflection and class imbalance issues. In this paper, an attention-guided network is proposed to segment the cataract surgical instrument. A new attention module is designed to learn discriminative features and address the specular reflection issue. It captures global context and encodes semantic dependencies to emphasize key semantic features, boosting the feature representation. This attention module…
Zhejiang University（浙江大学）；University of Notre Dame（圣母大学）
Surgery is the only viable treatment for cataract patients with visual acuity (VA) impairment. Clinically, to assess the necessity of cataract surgery, accurately predicting postoperative VA before surgery by analyzing multi-view optical coherence tomography (OCT) images is crucially needed. Unfortunately, due to complicated fundus conditions, determining postoperative VA remains difficult for medical experts. Deep learning methods for this problem were developed in recent years. Although effective, these methods still face several issues, such as not efficiently exploring potential relations …
Sterilizable Scene Graph Generation for Operating Rooms
Technical University of Darmstadt（达姆施塔特工业大学）；ImFusion GmbH（ImFusion有限公司）；Carl Zeiss AG（卡尔蔡司股份公司）；University Medical Center Mainz（美因茨大学医学中心）；Inria Center at University Côte d’Azur（蔚蓝海岸大学法国国家信息与自动化研究所中心）
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学 ICube 实验室、法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（A. Gemelli 大学综合医院基金会 IRCCS）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡 IHU 影像引导外科研究所）；Institute for Research against Digestive Cancer (IRCAD)（消化道癌症研究院（IRCAD））
A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets. This bottleneck is particularly prohibitive in highly specialized and regulated fields such as medicine and surgery, where video semantic segmentation could have important applications but data and expert annotations are scarce. In these settings, temporal clues and anatomical constraints could be leveraged during training to improve performance. Here, we present Temporally Constrained Neural Networks (TCNN), a semi-supervised fr…
Supervised training of neural networks requires large, diverse and well annotated data sets. In the medical field, this is often difficult to achieve due to constraints in time, expert knowledge and prevalence of an event. Artificial data augmentation can help to prevent overfitting and improve the detection of rare events as well as overall performance. However, most augmentation techniques use purely spatial transformations, which are not sufficient for video data with temporal correlations. In this paper, we present a novel methodology for workflow augmentation and demonstrate its benefit f…
Technische Universität München（慕尼黑工业大学）；Carl Zeiss Meditec AG（卡尔蔡司医疗技术公司）；Vienna Hanusch Hospital（维也纳哈努施医院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Ludwig-Maximilians Universität München（慕尼黑大学）
Today, cataract surgery is the most frequently performed ophthalmic surgery in the world. The cataract, a developing opacity of the human eye lens, constitutes the world's most frequent cause for blindness. During surgery, the lens is removed and replaced by an artificial intraocular lens (IOL). To prevent patients from needing strong visual aids after surgery, a precise prediction of the optical properties of the inserted IOL is crucial. There has been lots of activity towards developing methods to predict these properties from biometric eye data obtained by OCT devices, recently also by empl…
