Recognition of surgical activity is an essential component to develop context-aware decision support for the operating room. In this work, we tackle the recognition of fine-grained activities, modeled as action triplets <instrument, verb, target> representing the tool activity. To this end, we introduce a new laparoscopic dataset, CholecT40, consisting of 40 videos from the public dataset Cholec80 in which all frames have been annotated using 128 triplet classes. Furthermore, we present an approach to recognize these triplets directly from the video data. It relies on a module called Class Activation Guide (CAG), which uses the instrument activation maps to guide the verb and target recognition. To model the recognition of multiple triplets in the same frame, we also propose a trainable 3D Interaction Space, which captures the associations between the triplet components. Finally, we demonstrate the significance of these contributions via several ablation studies and comparisons to baselines on CholecT40.
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学ICube研究所，法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学综合医院基金会IRCCS）；IRCAD France（法国消化癌症研究研究所）
Out of all existing frameworks for surgical workflow analysis in endoscopic videos, action triplet recognition stands out as the only one aiming to provide truly fine-grained and comprehensive information on surgical activities. This information, presented as <instrument, verb, target> combinations, is highly challenging to be accurately identified. Triplet components can be difficult to recognize individually; in this task, it requires not only performing recognition simultaneously for all three triplet components, but also correctly establishing the data association between them. To achieve …
Purpose Automated segmentation of anatomical structures in medical image analysis is a prerequisite for autonomous diagnosis as well as various computer and robot aided interventions. Recent methods based on deep convolutional neural networks (CNN) have outperformed former heuristic methods. However, those methods were primarily evaluated on rigid, real-world environments. In this study, existing segmentation methods were evaluated for their use on a new dataset of transoral endoscopic exploration. Methods Four machine learning based methods SegNet, UNet, ENet and ErfNet were trained with supe…
Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensive surgical perception framework, Semantic-SuPer, that integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes, benefiting dow…
IBM Research - Europe（IBM欧洲研究院）；University College Dublin（都柏林大学学院）；Royal College of Surgeons（皇家外科医学院）
Intra-operative identification of malignant versus benign or healthy tissue is a major challenge in fluorescence guided cancer surgery. We propose a perfusion quantification method for computer-aided interpretation of subtle differences in dynamic perfusion patterns which can be used to distinguish between normal tissue and benign or malignant tumors intra-operatively in real-time by using multispectral endoscopic videos. The method exploits the fact that vasculature arising from cancer angiogenesis gives tumors differing perfusion patterns from the surrounding tissue, and defines a signature …
Politecnico di Milano（米兰理工大学）；Université de Strasbourg（斯特拉斯堡大学）；Hospital Papa Giovanni XXIII（教皇约翰二十三世医院）
Objective: Accurate visual classification of bladder tissue during Trans-Urethral Resection of Bladder Tumor (TURBT) procedures is essential to improve early cancer diagnosis and treatment. During TURBT interventions, White Light Imaging (WLI) and Narrow Band Imaging (NBI) techniques are used for lesion detection. Each imaging technique provides diverse visual information that allows clinicians to identify and classify cancerous lesions. Computer vision methods that use both imaging techniques could improve endoscopic diagnosis. We address the challenge of tissue classification when annotation…
Registration-Grounded Spectral Fusion for Unregistered WLI/NBI Endoscopic Lesion Segmentation
The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）；School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Robust Real-Time Endoscopic Stereo Matching under Fuzzy Tissue Boundaries
The School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China（生物医学工程学院，上海交通大学）；Innovation Center for Electronic Design Automation Technology, Hangzhou Dianzi University, Hangzhou 310018, China（电子设计自动化技术创新中心，杭州电子科技大学）
SurgT challenge: Benchmark of Soft-Tissue Trackers for Robotic Surgery
Imperial College London（伦敦帝国理工学院）；Jmees（Jmees公司）；Electronics and Telecommunications Research Institute (ETRI)（电子通信研究院）；Ajou University（亚洲大学）；University of Toronto（多伦多大学）；University College London（伦敦大学学院）
TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation
Birmingham City University（伯明翰城市大学）；University Hospitals Birmingham（伯明翰大学医院）；University of Birmingham（伯明翰大学）；University of the West of England（西英格兰大学）；University of Hertfordshire（赫特福德大学）；University of Bristol（布里斯托大学）；Qatar University（卡塔尔大学）
National University of Sciences and Technology（国家科技大学）；National Center for Artificial Intelligence（国家人工智能中心）
Autonomous surgical procedures, in particular minimal invasive surgeries, are the next frontier for Artificial Intelligence research. However, the existing challenges include precise identification of the human anatomy and the surgical settings, and modeling the environment for training of an autonomous agent. To address the identification of human anatomy and the surgical settings, we propose a deep learning based semantic segmentation algorithm to identify and label the tissues and organs in the endoscopic video feed of the human torso region. We present an annotated dataset, m2caiSeg, creat…
CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Technical University Munich（慕尼黑工业大学）；Riwolink GmbH（Riwolink有限公司）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Shanghai Jiao Tong University（上海交通大学）；IPCA（葡萄牙卡瓦多和阿韦河理工学院）；University of Minho（米尼奥大学）
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
EndoControlMag: Robust Endoscopic Vascular Motion Magnification with Periodic Reference Resetting and Hierarchical Tissue-aware Dual-Mask Control
organization= Department of Electronic Engineering, The Chinese University of Hong Kong , state= Hong Kong SAR , country= China；organization= The Chinese University of Hong Kong Shenzhen Research Institute , city= Shen Zhen , state= Guangdong , country= China；organization= The First Affiliated Hospital of Wenzhou Medical University , city= Wenzhou , state= Zhejiang , country= China；organization= Department of General Surgery \& Guangdong Provincial Key Laboratory of Precision Medicine for Gastrointestinal Tumor, Nanfang Hospital, Southern Medical University , city= Guang Zhou , state= Guangdong , country= China；organization= Department of Otolaryngology-Head；Neck Surgery, Singapore General Hospital, Duke-NUS Medical School , country= Singapore；organization= Shenzhen Key Laboratory of Robotics Perception；the Department of Electronic；Electrical Engineering, Southern University of Science
A Graph-Augmented knowledge Distillation based Dual-Stream Vision Transformer with Region-Aware Attention for Gastrointestinal Disease Classification with Explainable AI
Department of Computer Science and Engineering（计算机科学与工程系）
Politecnico di Milano（米兰理工大学）；Istituto Italiano di Tecnologia（意大利技术研究院）；German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Heidelberg University Hospital（海德堡大学医院）
Objective: Surgical data science is evolving into a research field that aims to observe everything occurring within and around the treatment process to provide situation-aware data-driven assistance. In the context of endoscopic video analysis, the accurate classification of organs in the field of view of the camera proffers a technical challenge. Herein, we propose a new approach to anatomical structure classification and image tagging that features an intrinsic measure of confidence to estimate its own performance with high reliability and which can be applied to both RGB and multispectral i…
CholecTriplet2021: A benchmark challenge for surgical action triplet recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Chicago（芝加哥大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Hefei University of Technology（合肥工业大学）；Shanghai Jiao Tong University（上海交通大学）；Argonne National Laboratory（阿贡国家实验室）；Medtronic（美敦力）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Reconstruction of optical vector-fields with applications in endoscopic imaging
University of Cambridge（剑桥大学）；Statistical Laboratory, University of Cambridge（剑桥大学统计实验室）；Department of Engineering, University of Cambridge（剑桥大学工程系）；Instituto de Telecomunicações, Faculdade de Ciências da Universidade do Porto（波尔图大学理学院电信研究所）；Nokia Bell Labs, Cambridge（诺基亚贝尔实验室（剑桥））；Wellcome Trust-MRC Cambridge Stem Cell Institute, University of Cambridge（剑桥大学惠康-医学研究委员会剑桥干细胞研究所）；Department of Physics and Cancer Research UK Cambridge Institute, University of Cambridge（剑桥大学物理系及癌症研究英国剑桥研究所）
SurgiSR4K: A High-Resolution Endoscopic Video Dataset for Robotic-Assisted Minimally Invasive Procedures
Intuitive Surgical, Inc.（Intuitive Surgical公司）；Johns Hopkins Medicine Neurosurgery（约翰霍普金斯医学神经外科）；Johns Hopkins University Electrical and Computer Engineering（约翰霍普金斯大学电气与计算机工程）；University of British Columbia Electrical and Computer Engineering（不列颠哥伦比亚大学电气与计算机工程）；Wilford & Kate Bailey Small Animal Teaching Hospital（威尔福德与凯蒂·贝利小动物教学医院）
Digital Surgery Ltd, Medtronic（美敦力数字外科有限公司）；ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube实验室、法国国家科学研究中心、斯特拉斯堡大学医院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hutom（Hutom公司）；Xiamen University（厦门大学）；University of Electronic Science and Technology of China（电子科技大学）；National University of Singapore（新加坡国立大学）；National Institute of Technology, Raipur（赖布尔国家理工学院）；King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
Surgical scene segmentation is essential for anatomy and instrument localization which can be further used to assess tissue-instrument interactions during a surgical procedure. In 2017, the Challenge on Automatic Tool Annotation for cataRACT Surgery (CATARACTS) released 50 cataract surgery videos accompanied by instrument usage annotations. These annotations included frame-level instrument presence information. In 2020, we released pixel-wise semantic annotations for anatomy and instruments for 4670 images sampled from 25 videos of the CATARACTS training set. The 2020 CATARACTS Semantic Segmen…
