CholecTriplet2021: A benchmark challenge for surgical action triplet recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Chicago（芝加哥大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Hefei University of Technology（合肥工业大学）；Shanghai Jiao Tong University（上海交通大学）；Argonne National Laboratory（阿贡国家实验室）；Medtronic（美敦力）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Context-aware decision support in the operating room can foster surgical safety and efficiency by leveraging real-time feedback from surgical workflow analysis. Most existing works recognize surgical activities at a coarse-grained level, such as phases, steps or events, leaving out fine-grained interaction details about the surgical activity; yet those are needed for more helpful AI assistance in the operating room. Recognizing surgical actions as triplets of <instrument, verb, target> combination delivers comprehensive details about the activities taking place in surgical videos. This paper presents CholecTriplet2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. The challenge granted private access to the large-scale CholecT50 dataset, which is annotated with action triplet information. In this paper, we present the challenge setup and assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods from the challenge organizers and 19 new deep learning algorithms by competing teams are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%. This study also analyzes the significance of the results obtained by the presented approaches, performs a thorough methodological comparison between them, in-depth result analysis, and proposes a novel ensemble method for enhanced recognition. Our analysis shows that surgical workflow analysis is not yet solved, and also highlights interesting directions for future research on fine-grained surgical activity recognition which is of utmost importance for the development of AI in surgery.
ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection
School of Nursing, The Hong Kong Polytechnic University（香港理工大学护理学院）；Nanfang Hospital, Southern Medical University（南方医科大学南芳医院）；Department of Surgery, The Chinese University of Hong Kong（香港中文大学外科部）；South China University of Technology（华南理工大学）；Singapore Management University（新加坡国立管理学院）
CNRS（法国国家科学研究中心）；University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）
In addition to generating data and annotations, devising sensible data splitting strategies and evaluation metrics is essential for the creation of a benchmark dataset. This practice ensures consensus on the usage of the data, homogeneous assessment, and uniform comparison of research methods on the dataset. This study focuses on CholecT50, which is a 50 video surgical dataset that formalizes surgical activities as triplets of <instrument, verb, target>. In this paper, we introduce the standard splits for the CholecT50 and CholecT45 datasets and show how they compare with existing use of the d…
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学ICube研究所，法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学综合医院基金会IRCCS）；IRCAD France（法国消化癌症研究研究所）
Out of all existing frameworks for surgical workflow analysis in endoscopic videos, action triplet recognition stands out as the only one aiming to provide truly fine-grained and comprehensive information on surgical activities. This information, presented as <instrument, verb, target> combinations, is highly challenging to be accurately identified. Triplet components can be difficult to recognize individually; in this task, it requires not only performing recognition simultaneously for all three triplet components, but also correctly establishing the data association between them. To achieve …
University of Rennes 1（雷恩第一大学）；INSERM（法国国家健康与医学研究院）；Gazi University（加齐大学）
Modeling and recognition of surgical activities poses an interesting research problem. Although a number of recent works studied automatic recognition of surgical activities, generalizability of these works across different tasks and different datasets remains a challenge. We introduce a modality that is robust to scene variation, and that is able to infer part information such as orientational and relative spatial relationships. The proposed modality is based on spatial temporal graph representations of surgical tools in videos, for surgical activity recognition. To explore its effectiveness,…
Automatic surgical activity recognition enables more intelligent surgical devices and a more efficient workflow. Integration of such technology in new operating rooms has the potential to improve care delivery to patients and decrease costs. Recent works have achieved a promising performance on surgical activity recognition; however, the lack of generalizability of these models is one of the critical barriers to the wide-scale adoption of this technology. In this work, we study the generalizability of surgical activity recognition models across operating rooms. We propose a new domain adaptati…
Activity recognition in surgical videos is a key research area for developing next-generation devices and workflow monitoring systems. Since surgeries are long processes with highly-variable lengths, deep learning models used for surgical videos often consist of a two-stage setup using a backbone and temporal sequence model. In this paper, we investigate many state-of-the-art backbones and temporal models to find architectures that yield the strongest performance for surgical activity recognition. We first benchmark the models performance on a large-scale activity recognition dataset containin…
Surgical Workflow Recognition and Blocking Effectiveness Detection in Laparoscopic Liver Resections with Pringle Maneuver
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Nanfang Hospital, Southern Medical University（南方医科大学南方医院）
Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)
Regensburg Medical Image Computing (ReMIC), OTH Regensburg（雷根斯堡医学影像计算中心（ReMIC））；Regensburg Center of Biomedical Engineering (RCBE), OTH Regensburg（雷根斯堡生物医学工程中心（RCBE））；Regensburg University（雷根斯堡大学）；Regensburg Center of Health Sciences（雷根斯堡健康科学中心）；Technical University of Munich (TUM)（慕尼黑技术大学）；TUM University Hospital（慕尼黑技术大学医院）；Biomedical Image Analysis Group, Department of Computing, Imperial College London（生物医学影像分析组，计算部门，伦敦帝国学院）；Research Group MITI, TUM University Hospital, School of Medicine（MITI研究组，慕尼黑技术大学医院，医学学院）；Department of Surgery, TUM University Hospital, School of Medicine（外科部门，慕尼黑技术大学医院，医学学院）
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）
Surgical gesture recognition is important for surgical data science and computer-aided intervention. Even with robotic kinematic information, automatically segmenting surgical steps presents numerous challenges because surgical demonstrations are characterized by high variability in style, duration and order of actions. In order to extract discriminative features from the kinematic signals and boost recognition accuracy, we propose a multi-task recurrent neural network for simultaneous recognition of surgical gestures and estimation of a novel formulation of surgical task progress. To show the…
TUNeS: A Temporal U-Net with Self-Attention for Video-based Surgical Phase Recognition
Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC) Dresden（转化外科肿瘤学系，国家肿瘤中心（NCT/UCC）德累斯顿）；German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（医学院和卡尔·古斯塔夫·卡尔斯大学医院，德累斯顿技术大学）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗斯托克霍尔茨研究中心（HZDR））；Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology（具有人机交互的触觉互联网研究中心（CeTI），德累斯顿技术大学）；BMBF Research Hub 6G-Life, TUD Dresden University of Technology（德国联邦教育与研究部6G-Life研究 hub，德累斯顿技术大学）
Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition
The Hong Kong University of Science and Technology（香港科技大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
CurConMix+: A Unified Spatio-Temporal Framework for Hierarchical Surgical Workflow Understanding
Department of Medical Device Management and Research, Samsung Advanced Institute for Health Sciences & Technology (SAIHST), Sungkyunkwan University（医学院设备管理与研究系，三星健康科学与技术研究院（SAIHST），全北大学）；Clinical Robotics and Embodied AI Research Center, Smart Healthcare Research Institute, Research Institute for Future Medicine, Samsung Medical Center（临床机器人与具身人工智能研究中心，智能医疗研究机构，未来医学研究所，三星医疗中心）；Department of Surgery, Samsung Medical Center（外科系，三星医疗中心）
HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Technische Universität München（慕尼黑工业大学）
Geo-RepNet: Geometry-Aware Representation Learning for Surgical Phase Recognition in Endoscopic Submucosal Dissection
Department of Electronic Engineering, The Chinese University of Hong Kong（电子工程系，香港中文大学）；Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd.（理论实验室，中央研究院，2012实验室，华为技术有限公司）
OSSAR: Towards Open-Set Surgical Activity Recognition in Robot-assisted Surgery
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Beijing Institute of Technology（北京理工大学）；Qilu Hospital of Shandong University（山东大学齐鲁医院）；Tongji University（同济大学）；University College London（伦敦大学学院）
Heidelberg University Hospital（海德堡大学医院）；National Center for Tumor Diseases (NCT) Heidelberg（海德堡国家肿瘤疾病中心）；KARL STORZ SE & Co. KG（卡尔史托斯股份有限公司）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center dataset. In this work we investigated the generalizability of phase recognition algorithms in a multi-center setting including more difficult recognit…
Pixel-Wise Recognition for Holistic Surgical Scene Understanding
Center for Research and Formation in Artificial Intelligence (CinfonIA)（人工智能研究与形成中心（CinfonIA））；Universidad de los Andes（安第斯大学）；Fundación Santa Fe de Bogotá（圣菲波哥大基金会）
Self-Knowledge Distillation for Surgical Phase Recognition
Medtronic Digital Surgery（美敦力数字外科）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心）
Recognition of surgical gesture is crucial for surgical skill assessment and efficient surgery training. Prior works on this task are based on either variant graphical models such as HMMs and CRFs, or deep learning models such as Recurrent Neural Networks and Temporal Convolutional Networks. Most of the current approaches usually suffer from over-segmentation and therefore low segment-level edit scores. In contrast, we present an essentially different methodology by modeling the task as a sequential decision-making process. An intelligent agent is trained using reinforcement learning with hier…
University Hospital of Strasbourg（斯特拉斯堡大学医院）；IRCAD（斯特拉斯堡 IRCAD 研究所）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；ICube（斯特拉斯堡大学 ICube 实验室）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）
Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior. In laparoscopic procedures one particular algorithm needed for such systems is the identification of surgical phases, for which the current state of the art is a model based on a CNN-LSTM. A number of previous works using models of this kind have trained them in a fully supervised manner, requiring a fully annotated dataset. Instead, our work confronts the problem of learning surgical phase recognition in scenarios presenting scar…
Yanshan University（燕山大学）；Hebei Normal University of Science and Technology（河北科技师范学院）
With the widespread use of touch-screen devices, it is more and more convenient for people to draw sketches on screen. This results in the demand for automatically understanding the sketches. Thus, the sketch recognition task becomes more significant than before. To accomplish this task, it is necessary to solve the critical issue of improving the distinction of the sketch features. To this end, we have made efforts in three aspects. First, a novel multi-scale residual block is designed. Compared with the conventional basic residual block, it can better perceive multi-scale information and red…
Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition
ICube Laboratory（ICube实验室）；University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；CNRS（法国国家科学研究中心）
The Chinese University of Hong Kong（香港中文大学）；Imsight Medical Technology, Co, Ltd（Imsight医疗科技有限公司）；The Hong Kong Polytechnic University（香港理工大学）
Surgical tool presence detection and surgical phase recognition are two fundamental yet challenging tasks in surgical video analysis and also very essential components in various applications in modern operating rooms. While these two analysis tasks are highly correlated in clinical practice as the surgical process is well-defined, most previous methods tackled them separately, without making full use of their relatedness. In this paper, we present a novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simult…
Surgical action triplet recognition provides a better understanding of the surgical scene. This task is of high relevance as it provides the surgeon with context-aware support and safety. The current go-to strategy for improving performance is the development of new network mechanisms. However, the performance of current state-of-the-art techniques is substantially lower than other surgical tasks. Why is this happening? This is the question that we address in this work. We present the first study to understand the failure of existing deep learning models through the lens of robustness and expl…
Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions
ICube Laboratory（ICube实验室）；University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；CNRS（法国国家科学研究中心）
ICube（ICube实验室）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IRCAD（法国消化外科研究所）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学综合医院基金会IRCCS）
Recognition of surgical activity is an essential component to develop context-aware decision support for the operating room. In this work, we tackle the recognition of fine-grained activities, modeled as action triplets <instrument, verb, target> representing the tool activity. To this end, we introduce a new laparoscopic dataset, CholecT40, consisting of 40 videos from the public dataset Cholec80 in which all frames have been annotated using 128 triplet classes. Furthermore, we present an approach to recognize these triplets directly from the video data. It relies on a module called Class Act…
SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Pavia（帕维亚大学）；University Digestive Health Care Center – Clarunis（克拉鲁尼斯大学消化保健中心）；Università di Roma Tor Vergata（罗马第二大学）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；University Hospital Zurich（苏黎世大学医院）；University Hospital Bern（伯尔尼大学医院）；University of Basel（巴塞尔大学）
fine-CLIP: Enhancing Zero-Shot Fine-Grained Surgical Action Recognition with Vision-Language Models
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
TRCoRSurg: Temporal-Relational Co-Reasoning for Surgical Video Triplet Recognition
Beihang University（北京航空航天大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen University of Advanced Technology（深圳理工大学）
University of Strasbourg（斯特拉斯堡大学）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅里基金会天主教大学综合医院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Basel（巴塞尔大学）
Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis
Applied Robotics and AI Solutions (ARAS)（应用机器人与人工智能解决方案）；Faculties of Electrical and Computer Engineering（电气与计算机工程学院）；K.N. Toosi University of Technology（卡里姆·纳尼·托西技术大学）；Translational Ophthalmology Research Center（转化眼科研究中心）；Farabi Eye Hospital（法拉比眼科医院）；Tehran University of Medical Sciences（德黑兰医科大学）；Noor Ophthalmology Research Center（努尔眼科研究中心）；University of Alberta（阿尔伯塔大学）；Departments of Electrical and Computer Engineering & Biomedical Engineering（电气与计算机工程及生物医学工程系）
Automatic recognition of surgical phases in surgical videos is a fundamental task in surgical workflow analysis. In this report, we propose a Transformer-based method that utilizes calibrated confidence scores for a 2-stage inference pipeline, which dynamically switches between a baseline model and a separately trained transition model depending on the calibrated confidence level. Our method outperforms the baseline model on the Cholec80 dataset, and can be applied to a variety of action segmentation methods.
University of Rennes 1（雷恩第一大学）；Inserm（法国国家健康与医学研究院）
In this paper, we address the open research problem of surgical gesture recognition using motion cues from video data only. We adapt Optical flow ConvNets initially proposed by Simonyan et al.. While Simonyan uses both RGB frames and dense optical flow, we use only dense optical flow representations as input to emphasize the role of motion in surgical gesture recognition, and present it as a robust alternative to kinematic data. We also overcome one of the limitations of Optical flow ConvNets by initializing our model with cross modality pre-training. A large number of promising studies that a…
University of Hyogo（兵库大学）；Takatsuki Hospital（高槻医院）；International University of Business Agriculture & Technology（国际商业农业与技术大学）
Total knee arthroplasty (TKA) is a commonly performed surgical procedure to mitigate knee pain and improve functions for people with knee arthritis. The procedure is complicated due to the different surgical tools used in the stages of surgery. The recognition of surgical tools in real-time can be a solution to simplify surgical procedures for the surgeon. Also, the presence and movement of tools in surgery are crucial information for the recognition of the operational phase and to identify the surgical workflow. Therefore, this research proposes the development of a real-time system for the r…
Georgia Institute of Technology（佐治亚理工学院）；University of Southern California（南加利福尼亚大学）；Intuitive Surgical Inc.（直觉外科手术公司）
Adverse surgical outcomes are costly to patients and hospitals. Approaches to benchmark surgical care are often limited to gross measures across the entire procedure despite the performance of particular tasks being largely responsible for undesirable outcomes. In order to produce metrics from tasks as opposed to the whole procedure, methods to recognize automatically individual surgical tasks are needed. In this paper, we propose several approaches to recognize surgical activities in robot-assisted minimally invasive surgery using deep learning. We collected a clinical dataset of 100 robot-as…
HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Department of General Surgery, Beijing Friendship Hospital, Capital Medical University（首都医科大学附属北京友谊医院普通外科）；Chinese PLA General Hospital（中国人民解放军总医院）
ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube研究所、法国国家科学研究中心、斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg, IRCAD, IHU Strasbourg（斯特拉斯堡大学医院、斯特拉斯堡 IRCAD 研究所、斯特拉斯堡大学医院研究所）
Real-time algorithms for automatically recognizing surgical phases are needed to develop systems that can provide assistance to surgeons, enable better management of operating room (OR) resources and consequently improve safety within the OR. State-of-the-art surgical phase recognition algorithms using laparoscopic videos are based on fully supervised training. This limits their potential for widespread application, since creation of manual annotations is an expensive process considering the numerous types of existing surgeries and the vast amount of laparoscopic videos available. In this work…
The Hong Kong University of Science and Technology（香港科技大学）；Beihang University（北京航空航天大学）；Beihang Hangzhou Innovation Institute Yuhang（北航杭州创新研究院（余杭））
Surgical phase recognition is a fundamental task in computer-assisted surgery systems. Most existing works are under the supervision of expensive and time-consuming full annotations, which require the surgeons to repeat watching videos to find the precise start and end time for a surgical phase. In this paper, we introduce timestamp supervision for surgical phase recognition to train the models with timestamp annotations, where the surgeons are asked to identify only a single timestamp within the temporal boundary of a phase. This annotation can significantly reduce the manual annotation cost …
Surgical Temporal Action-aware Network with Sequence Regularization for Phase Recognition
Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science&Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）；Beijing Friendship Hospital, Capital Medical University（首都医科大学附属北京友谊医院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Wuhan AI Research（武汉人工智能研究院）；ObjectEye Inc.（奥博视科技公司）
