Scaling up self-supervised learning for improved surgical foundation models
Department of Electrical Engineering, Video Coding \& Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands；Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands；Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands；Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands；Department of Urology, Catharina Hospital, Eindhoven, The Netherlands
Tim J. M. Jaspers, Ronald L. P. D. de Jong, Yiping Li, Carolus H. J. Kusters, Franciscus H. A. Bakker, Romy C. van Jaarsveld, Gino M. Kuiper, Richard van Hilleg… 展开作者
Tim J. M. Jaspers, Ronald L. P. D. de Jong, Yiping Li, Carolus H. J. Kusters, Franciscus H. A. Bakker, Romy C. van Jaarsveld, Gino M. Kuiper, Richard van Hillegersberg, Jelle P. Ruurda, Willem M. Brinkman, Josien P. W. Pluim, Peter H. N. de With, Marcel Breeuwer, Yasmina Al Khalil, Fons van der Sommen
Foundation models have revolutionized computer vision by achieving vastly superior performance across diverse tasks through large-scale pretraining on extensive datasets. However, their application in surgical computer vision has been limited. This study addresses this gap by introducing SurgeNetXL, a novel surgical foundation model that sets a new benchmark in surgical computer vision. Trained on the largest reported surgical dataset to date, comprising over 4.7 million video frames, SurgeNetXL achieves consistent top-tier performance across six datasets spanning four surgical procedures and three tasks, including semantic segmentation, phase recognition, and critical view of safety (CVS) classification. Compared with the best-performing surgical foundation models, SurgeNetXL shows mean improvements of 2.4, 9.0, and 12.6 percent for semantic segmentation, phase recognition, and CVS classification, respectively. Additionally, SurgeNetXL outperforms the best-performing ImageNet-based variants by 14.4, 4.0, and 1.6 percent in the respective tasks. In addition to advancing model performance, this study provides key insights into scaling pretraining datasets, extending training durations, and optimizing model architectures specifically for surgical computer vision. These findings pave the way for improved generalizability and robustness in data-scarce scenarios, offering a comprehensive framework for future research in this domain. All models and a subset of the SurgeNetXL dataset, including over 2 million video frames, are publicly available at: https://github.com/TimJaspers0801/SurgeNet.
Surgical Anatomy Recognition with Context Learning using Foundation Representations
Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology（埃因霍温理工大学，生物医学工程系，医学图像分析）；Department of Electrical Engineering, Architectures for Reliable Image Analysis, Eindhoven University of Technology（埃因霍温理工大学，电气工程系，可靠图像分析架构）；Department of Surgery, University Medical Center Utrecht（乌得勒支大学医学中心，外科）；Department of Oncological Urology, University Medical Center Utrecht（乌得勒支大学医学中心，肿瘤泌尿外科）；Department of Urology, Catharina Hospital（凯瑟琳娜医院，泌尿外科）；Department of Electrical Engineering, Mobile Perception Systems Lab, Eindhoven University of Technology（埃因霍温理工大学，电气工程系，移动感知系统实验室）
SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentation
School of Biomedical Engineering and Imaging Sciences, King’s College London（生物医学工程与成像科学学院，伦敦国王学院）；College of Electronic and Information Engineering, Tongji University（电子与信息工程学院，同济大学）
Disentangling spatio-temporal knowledge for weakly supervised object detection and segmentation in surgical video
University of Pennsylvania（宾夕法尼亚大学）；Penn Computer Assisted Surgery and Outcomes Laboratory（宾夕法尼亚计算机辅助外科与结局实验室）；Department of Surgery, University of Pennsylvania（宾夕法尼亚大学外科系）；Department of Electrical and Systems Engineering, University of Pennsylvania（宾夕法尼亚大学电气与系统工程系）；Department of Computer and Information Science, University of Pennsylvania（宾夕法尼亚大学计算机与信息科学系）
VLM-Guided Iterative Refinement for Surgical Image Segmentation with Foundation Models
School of Automation Science and Engineering, South China University of Technology, Guangzhou, China（自动化科学与工程学院，华南理工大学，广州，中国）；Vanderbilt University, Nashville, TN, USA（范德比尔特大学，纳什维尔，田纳西州，美国）；The Pennsylvania State University, University Park, PA , USA（宾夕法尼亚州立大学，大学园，宾夕法尼亚州，美国）；Peking University, Beijing , China（北京大学，北京，中国）；Virginia Commonwealth University, Richmond, VA, USA（弗吉尼亚共同wealth大学，里士满，弗吉尼亚州，美国）；University of California San Diego, San Diego, CA, USA（加州大学圣地亚哥分校，圣地亚哥，加利福尼亚州，美国）
Surgical-DeSAM: Decoupling SAM for Instrument Segmentation in Robotic Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（Wellcome/EPSRC介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；Dept. of Computer Science, University College London（伦敦大学学院计算机科学系）
EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；School of Advanced Manufacturing, Fuzhou University（福州大学先进制造学院）
SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision
Department of Computing, Imperial College London（帝国理工学院计算系）；Hamlyn Centre for Robotic Surgery, Department of Surgery and Cancer, Imperial College London（帝国理工学院外科与癌症部门机器人手术中心）
Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data
Department of Computer Science and Engineering, University at Buffalo（计算机科学与工程系，布法罗大学）；Department of Surgery, University at Buffalo（外科系，布法罗大学）；Department of Computer Science and Engineering, IIT, Jodhpur（计算机科学与工程系，印度理工学院，乔杜尔）；Department of Computer Science and Engineering, University of Notre Dame（计算机科学与工程系，圣母大学）
Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning
NUS Graduate School for Integrative Sciences and Engineering (NGS)（新加坡国立大学综合科学与工程研究生院（NGS））；National University of Singapore（新加坡国立大学）；University of Moratuwa（莫拉图瓦大学）；NIT Trichy（印度国立理工学院蒂鲁吉拉伯利分校）
Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks
ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube研究所、法国国家科学研究中心、斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg, IRCAD, IHU Strasbourg（斯特拉斯堡大学医院、斯特拉斯堡 IRCAD 研究所、斯特拉斯堡大学医院研究所）
Scaling Video Pretraining for Surgical Foundation Models
The Johns Hopkins University（约翰霍普金斯大学）；Zhejiang University（浙江大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳-香槟分校联合学院）；Zhejiang Lab（之江实验室）；Shaoxing University（绍兴大学）；Angelalign（时代天使）；Harvard Medical School（哈佛医学院）
Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video
The Chinese University of Hong Kong（香港中文大学）；Shun Hing Institute of Advanced Engineering, CUHK（香港中文大学晨兴先进工程研究所）；T Stone Robotics Institute, CUHK（香港中文大学T Stone机器人研究所）
Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review
MBZ University of AI（MBZ人工智能大学）；Hamad Bin Khalifa University（哈马德·本·卡西姆大学）；Birmingham City University（伯明翰城市大学）；Qatar University（卡塔尔大学）；University Hospitals Birmingham and University of Birmingham（伯明翰大学医院和伯明翰大学）
Free Lunch for Surgical Video Understanding by Distilling Self-Supervisions
The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology Shenzhen Research Institute（香港科技大学深圳研究院）
Label-supervised surgical instrument segmentation using temporal equivariance and semantic continuity
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（中国科学技术大学生物医学工程学院，生命科学与医学系）；Center for Medical Imaging, Robotics, Analytic Computing & Learning(MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州先进研究院医学影像中心、机器人与分析计算与学习中心）；Key Laboratory of Precision and Intelligent Chemistry, University of Science and Technology of China（中国科学技术大学精密与智能化学重点实验室）；Key Lab of Intelligent Information Processing of Chinese Academy of Sciences(CAS), Institute of Computing Technology, CAS（中国科学院智能信息处理重点实验室，计算技术研究所）；Faculty of Hepato-Biliary-Pancreatic Surgery, The First Medical Center, Chinese PLA General Hospital（中国人民解放军总医院肝胆胰外科系）
Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心）；National University of Singapore（新加坡国立大学）
Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions
ICube Laboratory（ICube实验室）；University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；CNRS（法国国家科学研究中心）
Large-scale Self-supervised Video Foundation Model for Intelligent Surgery
The Hong Kong University of Science and Technology（香港科技大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；The Hong Kong Polytechnic University（香港理工大学）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）；The Chinese University of Hong Kong（香港中文大学）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Shenzhen-Hong Kong Collaborative Innovation Research Institute（深港协同创新研究院）
SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China（人工智能与机器人中心，香港科学与创新研究院，中国科学院，香港，中国）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院，北京，中国）；Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany（医学辅助程序，慕尼黑技术大学，德国慕尼黑）；Electronic Engineering Department, The Chinese University of Hong Kong, Hong Kong, China（电子工程系，香港中文大学，香港，中国）；Neuromedical Centre, Hong Kong University Shenzhen Hospital, Shenzhen, China（神经医学中心，香港大学深圳医院，深圳，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；Department of Respiratory Medicine, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China（呼吸科，中山大学附属第一医院，广州，中国）
ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection
School of Nursing, The Hong Kong Polytechnic University（香港理工大学护理学院）；Nanfang Hospital, Southern Medical University（南方医科大学南芳医院）；Department of Surgery, The Chinese University of Hong Kong（香港中文大学外科部）；South China University of Technology（华南理工大学）；Singapore Management University（新加坡国立管理学院）
Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis
Department of Biomedical Engineering, University of Basel, Allschwil, Switzerland（巴塞尔大学生物医学工程系）；Clarunis – University Digestive Health Care Center Basel, Basel, Switzerland（Clarunis – 巴塞尔大学消化健康医疗中心）
Distilling Expert Surgical Knowledge: How to train local surgical VLMs for anatomy explanation in Complete Mesocolic Excision
1 Institute of Medical Technology；Intelligent Systems, Hamburg University of Technology, Hamburg, Germany 2 Department of General, Visceral；Thoracic Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
Pseudo-label Guided Cross-video Pixel Contrast for Robotic Surgical Scene Segmentation with Limited Annotations
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University College London（伦敦大学学院）；Zhejiang Lab（之江实验室）
Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition
University of Verona（维罗纳大学）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IRCAD（斯特拉斯堡微创外科研究中心）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学医院基金会IRCCS）
