In cataract surgery, the operation is performed with the help of a microscope. Since the microscope enables watching real-time surgery by up to two people only, a major part of surgical training is conducted using the recorded videos. To optimize the training procedure with the video content, the surgeons require an automatic relevance detection approach. In addition to relevance-based retrieval, these results can be further used for skill assessment and irregularity detection in cataract surgery videos. In this paper, a three-module framework is proposed to detect and classify the relevant phase segments in cataract videos. Taking advantage of an idle frame recognition network, the video is divided into idle and action segments. To boost the performance in relevance detection, the cornea where the relevant surgical actions are conducted is detected in all frames using Mask R-CNN. The spatiotemporally localized segments containing higher-resolution information about the pupil texture and actions, and complementary temporal information from the same phase are fed into the relevance detection module. This module consists of four parallel recurrent CNNs being responsible to detect four relevant phases that have been defined with medical experts. The results will then be integrated to classify the action phases as irrelevant or one of four relevant phases. Experimental results reveal that the proposed approach outperforms static CNNs and different configurations of feature-based and end-to-end recurrent networks.
Cataract is caused due to various factors such as age, trauma, genetics, smoking and substance consumption, and radiation. It is one of the major common ophthalmic diseases worldwide which can potentially affect iris-based biometric systems. India, which hosts the largest biometrics project in the world, has about 8 million people undergoing cataract surgery annually. While existing research shows that cataract does not have a major impact on iris recognition, our observations suggest that the iris segmentation approaches are not well equipped to handle cataract or post cataract surgery cases.…
University College London (UCL)（伦敦大学学院）；King’s College London (KCL)（伦敦国王学院）；Moorfields Eye Hospital（摩尔菲尔兹眼科医院）；Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences（惠康/英国工程与物理科学研究委员会介入与外科科学中心）；School of Biomedical Engineering & Imaging Sciences（生物医学工程与影像科学学院）
Our work proposes neural network design choices that set the state-of-the-art on a challenging public benchmark on cataract surgery, CaDIS. Our methodology achieves strong performance across three semantic segmentation tasks with increasingly granular surgical tool class sets by effectively handling class imbalance, an inherent challenge in any surgical video. We consider and evaluate two conceptually simple data oversampling methods as well as different loss functions. We show significant performance gains across network architectures and tasks especially on the rarest tool classes, thereby p…
Inserm, UMR 1101（法国国家健康与医学研究院）；Univ Bretagne Occidentale（西布列塔尼大学）；INSTITUT TELECOM（法国电信学院；布雷斯特电信学院）；TELECOM Bretagne（布雷斯特大学医院）；CHRU Brest
The amount of surgical data, recorded during video-monitored surgeries, has extremely increased. This paper aims at improving existing solutions for the automated analysis of cataract surgeries in real time. Through the analysis of a video recording the operating table, it is possible to know which instruments exit or enter the operating table, and therefore which ones are likely being used by the surgeon. Combining these observations with observations from the microscope video should enhance the overall performance of the systems. To this end, the proposed solution is divided into two main pa…
Deep-Learning-Assisted Analysis of Cataract Surgery Videos
Alpen-Adria-Universität Klagenfurt（阿尔卑斯-亚得里亚大学克拉根福分校）；University of Applied Sciences Western Switzerland（瑞士西部应用科学大学）；University of Bern（伯尔尼大学）；University of Geneva（日内瓦大学）
INSTITUT MINES-TELECOM（法国国立高等矿业电信学校）；TELECOM Bretagne（布列塔尼电信学院）；UEB（布列塔尼西部大学）；LaTIM - INSERM UMR 1101（LaTIM - 法国国家健康与医学研究院 UMR 1101）；Univ Bretagne Occidentale（布列塔尼西大学）；CHRU Brest（布雷斯特大学医院）
The automatic analysis of the surgical process, from videos recorded during surgeries, could be very useful to surgeons, both for training and for acquiring new techniques. The training process could be optimized by automatically providing some targeted recommendations or warnings, similar to the expert surgeon's guidance. In this paper, we propose to reuse videos recorded and stored during cataract surgeries to perform the analysis. The proposed system allows to automatically recognize, in real time, what the surgeon is doing: what surgical phase or, more precisely, what surgical step he or s…
University of Bern（伯尔尼大学）；Inselspital, University Hospital, University of Bern（伯尔尼大学附属医院（因塞尔医院））；ARTORG Center, University of Bern（伯尔尼大学ARTORG中心）
Cataract surgery is a sight saving surgery that is performed over 10 million times each year around the world. With such a large demand, the ability to organize surgical wards and operating rooms efficiently is critical to delivery this therapy in routine clinical care. In this context, estimating the remaining surgical duration (RSD) during procedures is one way to help streamline patient throughput and workflows. To this end, we propose CataNet, a method for cataract surgeries that predicts in real time the RSD jointly with two influential elements: the surgeon's experience, and the current …
University of Bern（伯尔尼大学）；Alpen-Adria-Universität Klagenfurt（克拉根福阿尔卑斯-亚德里亚大学）
Semantic segmentation in cataract surgery has a wide range of applications contributing to surgical outcome enhancement and clinical risk reduction. However, the varying issues in segmenting the different relevant structures in these surgeries make the designation of a unique network quite challenging. This paper proposes a semantic segmentation network, termed DeepPyramid, that can deal with these challenges using three novelties: (1) a Pyramid View Fusion module which provides a varying-angle global view of the surrounding region centering at each pixel position in the input convolutional fe…
$\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training
Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute, Johns Hopkins University（约翰斯·霍普金斯大学威尔默眼科研究所）；Malone Center for Engineering in Healthcare, Johns Hopkins University（约翰斯·霍普金斯大学马龙医疗保健工程中心）
Semantic segmentation in cataract surgery has a wide range of applications contributing to surgical outcome enhancement and clinical risk reduction. However, the varying issues in segmenting the different relevant instances make the designation of a unique network quite challenging. This paper proposes a semantic segmentation network termed as DeepPyram that can achieve superior performance in segmenting relevant objects in cataract surgery videos with varying issues. This superiority mainly originates from three modules: (i) Pyramid View Fusion, which provides a varying-angle global view of t…
Explainable AI-Powered Framework for Video-Based Skill Assessment in Cataract Surgery
K.N. Toosi University of Technology（霍加托西大学）；Faculties of Electrical and Computer Engineering（电气与计算机工程学院）；Applied Robotics and AI Solutions (ARAS)（应用机器人与人工智能解决方案（ARAS））
A critical complication after cataract surgery is the dislocation of the lens implant leading to vision deterioration and eye trauma. In order to reduce the risk of this complication, it is vital to discover the risk factors during the surgery. However, studying the relationship between lens dislocation and its suspicious risk factors using numerous videos is a time-extensive procedure. Hence, the surgeons demand an automatic approach to enable a larger-scale and, accordingly, more reliable study. In this paper, we propose a novel framework as the major step towards lens irregularity detection…
Instrument-tissue Interaction Detection Framework for Surgical Video Understanding
National University of Singapore（新加坡国立大学）；Research Institute of Trustworthy Autonomous Systems（可信自主系统研究院）；Southern University of Science and Technology（南方科技大学）；Agency for Science, Technology, and Research（新加坡科技研究局）；Osaka University Graduate School of Medicine（大阪大学医学部）
CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery
Chair for Computer-Aided Medical Procedures, Technical University Munich, Germany（计算机辅助医疗程序研究所，慕尼黑技术大学，德国）；Carl Zeiss AG, Munich, Germany（蔡司股份公司，慕尼黑，德国）
Phase-Specific Augmented Reality Guidance for Microscopic Cataract Surgery Using Long-Short Spatiotemporal Aggregation Transformer
Shanghai Jiao Tong University（上海交通大学）；Xinhua Hospital Affiliated to Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属新华医院）；Rice University（莱斯大学）；University of Münster（明斯特大学）
Digital Surgery LTD（数字手术有限公司）；Inserm（法国国家健康与医学研究院）
Video feedback provides a wealth of information about surgical procedures and is the main sensory cue for surgeons. Scene understanding is crucial to computer assisted interventions (CAI) and to post-operative analysis of the surgical procedure. A fundamental building block of such capabilities is the identification and localization of surgical instruments and anatomical structures through semantic segmentation. Deep learning has advanced semantic segmentation techniques in the recent years but is inherently reliant on the availability of labelled datasets for model training. This paper introd…
Department of Information Technology, Alpen-Adria-Universität Klagenfurt（克恩滕州阿尔卑斯-亚得里亚大学信息技术系）；Department of Ophthalmology, Klinikum Klagenfurt（克拉根福医院眼科）
Semantic segmentation in surgical videos is a prerequisite for a broad range of applications towards improving surgical outcomes and surgical video analysis. However, semantic segmentation in surgical videos involves many challenges. In particular, in cataract surgery, various features of the relevant objects such as blunt edges, color and context variation, reflection, transparency, and motion blur pose a challenge for semantic segmentation. In this paper, we propose a novel convolutional module termed as \textit{ReCal} module, which can calibrate the feature maps by employing region intra-an…
CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation
Thomas Jefferson High School for Science and Technology（托马斯·杰弗逊高中科学与技术学校）；Mass Eye and Ear, Harvard Medical School（麻省眼耳研究所，哈佛医学院）；Harvard Ophthalmology AI Lab, Schepens Eye Research Institute of Mass Eye and Ear, Harvard Medical School（哈佛眼科学人工智能实验室，麻省眼耳研究所的谢普恩斯眼研究学院，哈佛医学院）
This paper investigates the automatic monitoring of tool usage during a surgery, with potential applications in report generation, surgical training and real-time decision support. Two surgeries are considered: cataract surgery, the most common surgical procedure, and cholecystectomy, one of the most common digestive surgeries. Tool usage is monitored in videos recorded either through a microscope (cataract surgery) or an endoscope (cholecystectomy). Following state-of-the-art video analysis solutions, each frame of the video is analyzed by convolutional neural networks (CNNs) whose outputs ar…
Zhejiang University（浙江大学）；University of Notre Dame（圣母大学）
Surgery is the only viable treatment for cataract patients with visual acuity (VA) impairment. Clinically, to assess the necessity of cataract surgery, accurately predicting postoperative VA before surgery by analyzing multi-view optical coherence tomography (OCT) images is crucially needed. Unfortunately, due to complicated fundus conditions, determining postoperative VA remains difficult for medical experts. Deep learning methods for this problem were developed in recent years. Although effective, these methods still face several issues, such as not efficiently exploring potential relations …
Digital Surgery Ltd, Medtronic（美敦力数字外科有限公司）；ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube实验室、法国国家科学研究中心、斯特拉斯堡大学医院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hutom（Hutom公司）；Xiamen University（厦门大学）；University of Electronic Science and Technology of China（电子科技大学）；National University of Singapore（新加坡国立大学）；National Institute of Technology, Raipur（赖布尔国家理工学院）；King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
Surgical scene segmentation is essential for anatomy and instrument localization which can be further used to assess tissue-instrument interactions during a surgical procedure. In 2017, the Challenge on Automatic Tool Annotation for cataRACT Surgery (CATARACTS) released 50 cataract surgery videos accompanied by instrument usage annotations. These annotations included frame-level instrument presence information. In 2020, we released pixel-wise semantic annotations for anatomy and instruments for 4670 images sampled from 25 videos of the CATARACTS training set. The 2020 CATARACTS Semantic Segmen…
StepAL: Step-aware Active Learning for Cataract Surgical Videos
Johns Hopkins University（约翰霍普金斯大学）；Wilmer Eye Institute（威尔默眼科研究所）；Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院）；Malone Center for Engineering in Healthcare（马洛中心医疗工程中心）
Dept. of Computer Science University of Oldenburg（奥尔登堡大学计算机科学系）
Automatic tool detection from surgical imagery has a multitude of useful applications, such as real-time computer assistance for the surgeon. Using the successful residual network architecture, a system that can distinguish 21 different tools in cataract surgery videos is created. The videos are provided as part of the 2017 CATARACTS challenge and pose difficulties found in many real-world datasets, for example a strong class imbalance. The construction of the detection system is guided by a wide array of experiments that explore different design decisions.
Tracking and localizing objects is a central problem in computer-assisted surgery. Optical coherence tomography (OCT) can be employed as an optical tracking system, due to its high spatial and temporal resolution. Recently, 3D convolutional neural networks (CNNs) have shown promising performance for pose estimation of a marker object using single volumetric OCT images. While this approach relied on spatial information only, OCT allows for a temporal stream of OCT image volumes capturing the motion of an object at high volumes rates. In this work, we systematically extend 3D CNNs to 4D spatio-t…
Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis
Applied Robotics and AI Solutions (ARAS)（应用机器人与人工智能解决方案）；Faculties of Electrical and Computer Engineering（电气与计算机工程学院）；K.N. Toosi University of Technology（卡里姆·纳尼·托西技术大学）；Translational Ophthalmology Research Center（转化眼科研究中心）；Farabi Eye Hospital（法拉比眼科医院）；Tehran University of Medical Sciences（德黑兰医科大学）；Noor Ophthalmology Research Center（努尔眼科研究中心）；University of Alberta（阿尔伯塔大学）；Departments of Electrical and Computer Engineering & Biomedical Engineering（电气与计算机工程及生物医学工程系）
Digital Surgery, Kinosis, Ltd（Kinosis有限公司数字外科）；University College London（伦敦大学学院）
Automated surgical workflow analysis and understanding can assist surgeons to standardize procedures and enhance post-surgical assessment and indexing, as well as, interventional monitoring. Computer-assisted interventional (CAI) systems based on video can perform workflow estimation through surgical instruments' recognition while linking them to an ontology of procedural phases. In this work, we adopt a deep learning paradigm to detect surgical instruments in cataract surgery videos which in turn feed a surgical phase inference recurrent network that encodes temporal aspects of phase steps wi…
Modeling and automatically recognizing surgical activities are fundamental steps toward automation in surgery and play important roles in providing timely feedback to surgeons. Accurately recognizing surgical activities in video poses a challenging problem that requires an effective means of learning both spatial and temporal dynamics. Human gaze and visual saliency carry important information about visual attention and can be used to extract more relevant features that better reflect these spatial and temporal dynamics. In this study, we propose to use human gaze with a spatio-temporal attent…
GLSFormer: Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos
Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute, Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院威尔默眼科研究所）；Malone Center for Engineering in Healthcare, Johns Hopkins University（约翰斯·霍普金斯大学马龙医疗工程中心）
OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding
Monash University（莫纳什大学）；Airdoc-Monash Research（Airdoc-莫纳什研究所）；Airdoc；Bosch Corporate Research（博世企业研究院）；Cornell University（康奈尔大学）；King’s College London（伦敦国王学院）；Institute for Infocomm Research, A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））；City University of Macau（澳门城市大学）；Eye Hospital, Wenzhou Medical University（温州医科大学附属眼视光医院）
Large-scale Self-supervised Video Foundation Model for Intelligent Surgery
The Hong Kong University of Science and Technology（香港科技大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；The Hong Kong Polytechnic University（香港理工大学）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）；The Chinese University of Hong Kong（香港中文大学）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Shenzhen-Hong Kong Collaborative Innovation Research Institute（深港协同创新研究院）
NUS Graduate School for Integrative Sciences and Engineering (NGS), National University of Singapore（新加坡国立大学综合科学与工程研究生院）；Dept. of Biomedical Engineering, National University of Singapore（新加坡国立大学生物医学工程系）；Dept. of Instrumentation and Control Engineering, NIT Trichy（印度国立理工学院蒂鲁奇拉帕利分校仪器与控制工程系）；Dept. of Otolaryngology, Singapore General Hospital（新加坡中央医院耳鼻喉科）
Representation learning of the task-oriented attention while tracking instrument holds vast potential in image-guided robotic surgery. Incorporating cognitive ability to automate the camera control enables the surgeon to concentrate more on dealing with surgical instruments. The objective is to reduce the operation time and facilitate the surgery for both surgeons and patients. We propose an end-to-end trainable Spatio-Temporal Multi-Task Learning (ST-MTL) model with a shared encoder and spatio-temporal decoders for the real-time surgical instrument segmentation and task-oriented saliency dete…
Real-Time Spatio-Temporal Reconstruction of Dynamic Endoscopic Scenes with 4D Gaussian Splatting
University of Liverpool, Liverpool, UK（利物浦大学）；Xi'an Jiaotong-Liverpool University, Suzhou, China（西安交通大学利物浦大学）；University of Cambridge, Cambridge, UK（剑桥大学）
Activity recognition in surgical videos is a key research area for developing next-generation devices and workflow monitoring systems. Since surgeries are long processes with highly-variable lengths, deep learning models used for surgical videos often consist of a two-stage setup using a backbone and temporal sequence model. In this paper, we investigate many state-of-the-art backbones and temporal models to find architectures that yield the strongest performance for surgical activity recognition. We first benchmark the models performance on a large-scale activity recognition dataset containin…
Adaptive transfer learning for surgical tool presence detection in laparoscopic videos through gradual freezing fine-tuning
Institutes of Innovation for Future Society（未来社会创新研究所）；Nagoya University（名古屋大学）；Department of Micro-Nano Mechanical Science and Engineering（微纳米机械科学与工程系）
ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube、法国国家科学研究中心、斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IRCAD（消化癌症外科研究所）；IHU Strasbourg（斯特拉斯堡大学医院研究所）
This work presents a novel approach for the early recognition of the type of a laparoscopic surgery from its video. Early recognition algorithms can be beneficial to the development of 'smart' OR systems that can provide automatic context-aware assistance, and also enable quick database indexing. The task is however ridden with challenges specific to videos belonging to the domain of laparoscopy, such as high visual similarity across surgeries and large variations in video durations. To capture the spatio-temporal dependencies in these videos, we choose as our model a combination of a Convolut…
Stanford University（斯坦福大学）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）
Open procedures represent the dominant form of surgery worldwide. Artificial intelligence (AI) has the potential to optimize surgical practice and improve patient outcomes, but efforts have focused primarily on minimally invasive techniques. Our work overcomes existing data limitations for training AI models by curating, from YouTube, the largest dataset of open surgical videos to date: 1997 videos from 23 surgical procedures uploaded from 50 countries. Using this dataset, we developed a multi-task AI model capable of real-time understanding of surgical behaviors, hands, and tools - the buildi…
PitVis-2023 Challenge: Workflow Recognition in videos of Endoscopic Pituitary Surgery
University College London（伦敦大学学院）；National Hospital for Neurology and Neurosurgery（国立神经学与神经外科医院）；Centre for AI and Robotics (CAIR) HKISI, CAS（中国科学院香港创新研究院人工智能与机器人中心）；Shanghai Jiao Tong University（上海交通大学）
Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations
University of Las Palmas de Gran Canaria（大加那利岛拉斯帕尔马斯大学）；Imperial College London（伦敦帝国理工学院）；University Hospital Doctor Negrin（内格林大学医院）；University Hospital Southampton（南安普顿大学医院）；University of Cambridge（剑桥大学）；Universidad Politecnica de Madrid（马德里理工大学）
Adhesions are an important cause of chronic pain following abdominal surgery. Recent developments in abdominal cine-MRI have enabled the non-invasive diagnosis of adhesions. Adhesions are identified on cine-MRI by the absence of sliding motion during movement. Diagnosis and mapping of adhesions improves the management of patients with pain. Detection of abdominal adhesions on cine-MRI is challenging from both a radiological and deep learning perspective. We focus on classifying presence or absence of adhesions in sagittal abdominal cine-MRI series. We experimented with spatio-temporal deep lea…
Estimating the forces acting between instruments and tissue is a challenging problem for robot-assisted minimally-invasive surgery. Recently, numerous vision-based methods have been proposed to replace electro-mechanical approaches. Moreover, optical coherence tomography (OCT) and deep learning have been used for estimating forces based on deformation observed in volumetric image data. The method demonstrated the advantage of deep learning with 3D volumetric data over 2D depth images for force estimation. In this work, we extend the problem of deep learning-based force estimation to 4D spatio-…
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学 ICube 实验室、法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（A. Gemelli 大学综合医院基金会 IRCCS）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡 IHU 影像引导外科研究所）；Institute for Research against Digestive Cancer (IRCAD)（消化道癌症研究院（IRCAD））
A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets. This bottleneck is particularly prohibitive in highly specialized and regulated fields such as medicine and surgery, where video semantic segmentation could have important applications but data and expert annotations are scarce. In these settings, temporal clues and anatomical constraints could be leveraged during training to improve performance. Here, we present Temporally Constrained Neural Networks (TCNN), a semi-supervised fr…
Purpose. Localizing structures and estimating the motion of a specific target region are common problems for navigation during surgical interventions. Optical coherence tomography (OCT) is an imaging modality with a high spatial and temporal resolution that has been used for intraoperative imaging and also for motion estimation, for example, in the context of ophthalmic surgery or cochleostomy. Recently, motion estimation between a template and a moving OCT image has been studied with deep learning methods to overcome the shortcomings of conventional, feature-based methods. Methods. We investi…
Using Spatio-Temporal Dual-Stream Network with Self-Supervised Learning for Lung Tumor Classification on Radial Probe Endobronchial Ultrasound Video
National Yang Ming Chiao Tung University（国立阳明交通大学）；National Taiwan University Cancer Center（台湾大学癌症中心）；National Taiwan University Hospital（台湾大学医院）；National Taiwan University Hsin-Chu Hospital（台湾大学新竹医院）
TiS-TSL: Image-Label Supervised Surgical Video Stereo Matching via Time-Switchable Teacher-Student Learning
Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology（华中科技大学光电实验室）；Wuhan United Imaging Surgical Co., Ltd.（武汉联合影像手术有限公司）
Xiamen University（厦门大学）；The Chinese University of Hong Kong（香港中文大学）；Zhongshan Hospital affiliated to Xiamen University（厦门大学附属中山医院）；The Hong Kong Polytechnic University（香港理工大学）
Performing a real-time and accurate instrument segmentation from videos is of great significance for improving the performance of robotic-assisted surgery. We identify two important clues for surgical instrument perception, including local temporal dependency from adjacent frames and global semantic correlation in long-range duration. However, most existing works perform segmentation purely using visual cues in a single frame. Optical flow is just used to model the motion between only two frames and brings heavy computational cost. We propose a novel dual-memory network (DMNet) to wisely relat…
