The field of surgical computer vision has undergone considerable breakthroughs in recent years with the rising popularity of deep neural network-based methods. However, standard fully-supervised approaches for training such models require vast amounts of annotated data, imposing a prohibitively high cost; especially in the clinical domain. Self-Supervised Learning (SSL) methods, which have begun to gain traction in the general computer vision community, represent a potential solution to these annotation costs, allowing to learn useful representations from only unlabeled data. Still, the effectiveness of SSL methods in more complex and impactful domains, such as medicine and surgery, remains limited and unexplored. In this work, we address this critical need by investigating four state-of-the-art SSL methods (MoCo v2, SimCLR, DINO, SwAV) in the context of surgical computer vision. We present an extensive analysis of the performance of these methods on the Cholec80 dataset for two fundamental and popular tasks in surgical context understanding, phase recognition and tool presence detection. We examine their parameterization, then their behavior with respect to training data quantities in semi-supervised settings. Correct transfer of these methods to surgery, as described and conducted in this work, leads to substantial performance gains over generic uses of SSL - up to 7.4% on phase recognition and 20% on tool presence detection - as well as state-of-the-art semi-supervised phase recognition approaches by up to 14%. Further results obtained on a highly diverse selection of surgical datasets exhibit strong generalization properties. The code is available at https://github.com/CAMMA-public/SelfSupSurg.
Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data
Department of Computer Science and Engineering, University at Buffalo(计算机科学与工程系,布法罗大学);Department of Surgery, University at Buffalo(外科系,布法罗大学);Department of Computer Science and Engineering, IIT, Jodhpur(计算机科学与工程系,印度理工学院,乔杜尔);Department of Computer Science and Engineering, University of Notre Dame(计算机科学与工程系,圣母大学)
Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks
ICube, University of Strasbourg, CNRS, IHU Strasbourg(斯特拉斯堡大学ICube研究所、法国国家科学研究中心、斯特拉斯堡大学医院研究所);University Hospital of Strasbourg, IRCAD, IHU Strasbourg(斯特拉斯堡大学医院、斯特拉斯堡 IRCAD 研究所、斯特拉斯堡大学医院研究所)
Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video
The Chinese University of Hong Kong(香港中文大学);Shun Hing Institute of Advanced Engineering, CUHK(香港中文大学晨兴先进工程研究所);T Stone Robotics Institute, CUHK(香港中文大学T Stone机器人研究所)
Free Lunch for Surgical Video Understanding by Distilling Self-Supervisions
The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);The Hong Kong University of Science and Technology Shenzhen Research Institute(香港科技大学深圳研究院)
Label-supervised surgical instrument segmentation using temporal equivariance and semantic continuity
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China(中国科学技术大学生物医学工程学院,生命科学与医学系);Center for Medical Imaging, Robotics, Analytic Computing & Learning(MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏州先进研究院医学影像中心、机器人与分析计算与学习中心);Key Laboratory of Precision and Intelligent Chemistry, University of Science and Technology of China(中国科学技术大学精密与智能化学重点实验室);Key Lab of Intelligent Information Processing of Chinese Academy of Sciences(CAS), Institute of Computing Technology, CAS(中国科学院智能信息处理重点实验室,计算技术研究所);Faculty of Hepato-Biliary-Pancreatic Surgery, The First Medical Center, Chinese PLA General Hospital(中国人民解放军总医院肝胆胰外科系)
Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions
ICube Laboratory(ICube实验室);University of Strasbourg(斯特拉斯堡大学);IHU Strasbourg(斯特拉斯堡大学医院研究所);University Hospital of Strasbourg(斯特拉斯堡大学医院);CNRS(法国国家科学研究中心)
Large-scale Self-supervised Video Foundation Model for Intelligent Surgery
The Hong Kong University of Science and Technology(香港科技大学);German Cancer Research Center (DKFZ)(德国癌症研究中心);Heidelberg University Hospital(海德堡大学医院);The Hong Kong Polytechnic University(香港理工大学);National University of Singapore(新加坡国立大学);Tongji University(同济大学);The Chinese University of Hong Kong(香港中文大学);Heidelberg University(海德堡大学);National Center for Tumor Diseases (NCT)(国家肿瘤疾病中心);Shenzhen-Hong Kong Collaborative Innovation Research Institute(深港协同创新研究院)
ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection
School of Nursing, The Hong Kong Polytechnic University(香港理工大学护理学院);Nanfang Hospital, Southern Medical University(南方医科大学南芳医院);Department of Surgery, The Chinese University of Hong Kong(香港中文大学外科部);South China University of Technology(华南理工大学);Singapore Management University(新加坡国立管理学院)
SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentation
School of Biomedical Engineering and Imaging Sciences, King’s College London(生物医学工程与成像科学学院,伦敦国王学院);College of Electronic and Information Engineering, Tongji University(电子与信息工程学院,同济大学)
Scaling up self-supervised learning for improved surgical foundation models
Department of Electrical Engineering, Video Coding \& Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands;Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands;Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands;Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands;Department of Urology, Catharina Hospital, Eindhoven, The Netherlands
Disentangling spatio-temporal knowledge for weakly supervised object detection and segmentation in surgical video
University of Pennsylvania(宾夕法尼亚大学);Penn Computer Assisted Surgery and Outcomes Laboratory(宾夕法尼亚计算机辅助外科与结局实验室);Department of Surgery, University of Pennsylvania(宾夕法尼亚大学外科系);Department of Electrical and Systems Engineering, University of Pennsylvania(宾夕法尼亚大学电气与系统工程系);Department of Computer and Information Science, University of Pennsylvania(宾夕法尼亚大学计算机与信息科学系)
Pseudo-label Guided Cross-video Pixel Contrast for Robotic Surgical Scene Segmentation with Limited Annotations
The Chinese University of Hong Kong(香港中文大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University College London(伦敦大学学院);Zhejiang Lab(之江实验室)
Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition
University of Verona(维罗纳大学);University of Strasbourg(斯特拉斯堡大学);CNRS(法国国家科学研究中心);IHU Strasbourg(斯特拉斯堡大学医院研究所);University Hospital of Strasbourg(斯特拉斯堡大学医院);IRCAD(斯特拉斯堡微创外科研究中心);Fondazione Policlinico Universitario Agostino Gemelli IRCCS(阿戈斯蒂诺·杰梅利大学医院基金会IRCCS)
A semi-supervised Teacher-Student framework for surgical tool detection and localization
Tecnologico de Monterrey(蒙特雷理工学院);University of Leeds(利兹大学)
微创手术中的手术工具检测是计算机辅助干预的重要组成部分。目前的方法大多是基于监督的方法,需要大量的完全标记的数据来训练监督模型,并遭受伪标签偏见,因为类不平衡的问题。然而,带有限界框注释的大型图像数据集通常很少可用。半监督学习(SSL)最近已经成为一种仅使用少量注释数据来训练大型模型的方法;除了降低注释成本之外。SSL还显示出产生更健壮和更通用的模型的希望。因此,在本文中,我们介绍了一个半监督学习(SSL)框架的手术工具检测范例,旨在减轻训练数据的稀缺性和数据的不平衡,通过知识蒸馏的方法。在所提出的工作中,我们训练一个模型与标记的数据,初始化教师-学生联合学习,其中学生是训练教师生成的伪标签从未标记的数据。我们提出了一个多类距离与基于边缘的分类损失函数在感兴趣的区域头部的检测器,以有效地分离前景类从背景区域。我们在m2 cai 16-tool-locations数据集上的结果表明,我们的方法在不同的监督数据设置(1%,2%,5%,10%的注释数据)上的优越性,其中我们的模型在mAP(1%的标记数据)上分别比最先进的SSL方法和完全监督的基线实现了8%,12%和27%的整体改进。代码可在https://github.com/Mansoor-at/Semi-supervised-surgical-tool-det上获得
A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging
School of Biomedical Engineering & Imaging Sciences King’s College London London, UK(生物医学工程与成像科学学院 英国伦敦国王学院);Department of Neurosurgery King’s College Hospital NHS Foundation Trust, London, UK(神经外科部门 英国伦敦国王学院医院 NHS基金会信托)
SegMatch: A semi-supervised learning method for surgical instrument segmentation
School of Biomedical Engineering & Imaging Sciences, King’s College London(生物医学工程与影像科学学院,伦敦国王学院);Harvard Medical School, Harvard University(哈佛医学院,哈佛大学);College of Electronic and Information Engineering, Tongji University(同济大学电子与信息工程学院)
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France(斯特拉斯堡大学、法国国家科学研究中心、法国国家卫生研究院、ICube、UMR7357);IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France(斯特拉斯堡IHU、影像引导手术研究所);Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy(阿戈斯蒂诺·杰梅利大学附属医院基金会、意大利罗马);Azienda Ospedaliero-Universitaria Sant’Andrea, Rome, Italy(罗马大学附属医院);Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico di Milano, University of Milan, Milan, Italy(米兰Ca’ Granda主要医院基金会、米兰大学);Monaldi Hospital, AORN dei Colli, Naples, Italy(蒙塔尔迪医院、那不勒斯AORN dei Colli)
Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance
Surgical Artificial Intelligence Laboratory, Massachusetts General Hospital, MA, USA(马萨诸塞州总医院手术人工智能实验室);Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, MA, USA(麻省理工学院计算机科学与人工智能实验室);Department of Surgery, Duke University, NC, USA(达特茅斯大学外科系)
Weakly-Supervised Learning for Tool Localization in Laparoscopic Videos
ICube(ICube研究所);University of Strasbourg(斯特拉斯堡大学);CNRS(法国国家科学研究中心);IHU Strasbourg(斯特拉斯堡大学医院研究所);University Hospital of Strasbourg(斯特拉斯堡大学医院);IRCAD(消化癌症研究与治疗研究所)
Slot-BERT: Self-supervised Object Discovery in Surgical Video
Outcomes Laboratory, Department of Surgery, University of Pennsylvania, Philadelphia, PA, USA.;Department of Computer;Information Science, University of Pennsylvania, Philadelphia, PA, USA.;Division of Thoracic Surgery, Toronto General Hospital, University Health Network, Toronto, Ontario, Canada.;Surgical Artificial Intelligence Research Academy, University Health Network, Toronto, ON, Canada.
Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos
University Hospital of Strasbourg(斯特拉斯堡大学医院);IRCAD(法国消化外科研究所);IHU Strasbourg(斯特拉斯堡大学医院研究所);ICube(斯特拉斯堡计算机与图像科学实验室);University of Strasbourg(斯特拉斯堡大学);CNRS(法国国家科学研究中心)
Temporal coherence-based self-supervised learning for laparoscopic workflow analysis
National Center for Tumor Diseases (NCT)(国家肿瘤疾病中心(NCT));Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden(德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯医学院与大学医院)
SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation
The Hamlyn Centre for Robotic Surgery, Imperial College London(机器人手术中心,帝国理工学院伦敦分校);S-Lab, College of Computing and Data Science, Nanyang Technological University(S实验室, computing and Data Science学院,南洋理工大学);Department of Computer Science, University of Liverpool(计算机科学系,利物浦大学)