A generalizable foundation model for intraoperative understanding across surgical procedures
Kanggil Park, Yongjun Jeon, Soyoung Lim, Seonmin Park, Jongmin Shin, Jung Yong Kim, Sehyeon An, Jinsoo Rhu, Jongman Kim, Gyu-Seong Choi, Namkee Oh, Kyu-Hwan Jung
In minimally invasive surgery, clinical decisions depend on real-time visual interpretation, yet intraoperative perception varies substantially across surgeons and procedures. This variability limits consistent assessment, training, and the development of reliable artificial intelligence systems, as most surgical AI models are designed for narrowly defined tasks and do not generalize across procedures or institutions. Here we introduce ZEN, a generalizable foundation model for intraoperative surgical video understanding trained on more than 4 million frames from over 21 procedures using a self-supervised multi-teacher distillation framework. We curated a large and diverse dataset and systematically evaluated multiple representation learning strategies within a unified benchmark. Across 20 downstream tasks and full fine-tuning, frozen-backbone, few-shot and zero-shot settings, ZEN consistently outperforms existing surgical foundation models and demonstrates robust cross-procedure generalization. These results suggest a step toward unified representations for surgical scene understanding and support future applications in intraoperative assistance and surgical training assessment.
Surgical Workflow Recognition and Blocking Effectiveness Detection in Laparoscopic Liver Resections with Pringle Maneuver
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Nanfang Hospital, Southern Medical University（南方医科大学南方医院）
University of Rennes 1（雷恩第一大学）；INSERM（法国国家健康与医学研究院）；Gazi University（加齐大学）
Modeling and recognition of surgical activities poses an interesting research problem. Although a number of recent works studied automatic recognition of surgical activities, generalizability of these works across different tasks and different datasets remains a challenge. We introduce a modality that is robust to scene variation, and that is able to infer part information such as orientational and relative spatial relationships. The proposed modality is based on spatial temporal graph representations of surgical tools in videos, for surgical activity recognition. To explore its effectiveness,…
University of Michigan（密歇根大学）；Invenio Imaging（Invenio成像公司）；New York University（纽约大学）
Accurate intraoperative diagnosis is essential for providing safe and effective care during brain tumor surgery. Our standard-of-care diagnostic methods are time, resource, and labor intensive, which restricts access to optimal surgical treatments. To address these limitations, we propose an alternative workflow that combines stimulated Raman histology (SRH), a rapid optical imaging method, with deep learning-based automated interpretation of SRH images for intraoperative brain tumor diagnosis and real-time surgical decision support. Here, we present OpenSRH, the first public dataset of clinic…
Intraoperative Glioma Segmentation with YOLO + SAM for Improved Accuracy in Tumor Resection
Carnegie Vanguard High School（卡内基先锋高中）；Devon Preparatory High School（德文预备高中）；Lake Travis High School（特拉维斯湖高中）；Irvington High School（欧文顿高中）；Algoverse AI Research（Algoverse AI研究机构）
ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection
School of Nursing, The Hong Kong Polytechnic University（香港理工大学护理学院）；Nanfang Hospital, Southern Medical University（南方医科大学南芳医院）；Department of Surgery, The Chinese University of Hong Kong（香港中文大学外科部）；South China University of Technology（华南理工大学）；Singapore Management University（新加坡国立管理学院）
BridgeSplat: Bidirectionally Coupled CT and Non-Rigid Gaussian Splatting for Deformable Intraoperative Surgical Navigation
Computer Aided Medical Procedures, TU Munich, Munich, Germany（慕尼黑技术大学计算机辅助医疗程序部）；Department of General, Visceral, and Transplantation Surgery, Hospital of the LMU Munich, Ludwig-Maximilians-Universität (LMU), Munich, Germany（慕尼黑路德维希-马克西米利安大学（LMU）普通外科、 visceral外科及移植外科部门）；Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA（哈佛医学院布里特妇女医院）
Large-scale Self-supervised Video Foundation Model for Intelligent Surgery
The Hong Kong University of Science and Technology（香港科技大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；The Hong Kong Polytechnic University（香港理工大学）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）；The Chinese University of Hong Kong（香港中文大学）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Shenzhen-Hong Kong Collaborative Innovation Research Institute（深港协同创新研究院）
Early Operative Difficulty Assessment in Laparoscopic Cholecystectomy via Snapshot-Centric Video Analysis
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（法国斯特拉斯堡大学，CNRS，INSERM，ICube，UMR7357）；IHU Strasbourg, Strasbourg, France（斯特拉斯堡IHU）；Università degli Studi di Milano General Surgery Department（米兰大学外科部）；University of Torino（都灵大学）；Department of Medical Surgical Science and Translational Medicine, Sant’Andrea Hospital, Sapienza University of Rome（罗马Sapienza大学医学与外科科学及转化医学系，圣安德烈医院）；Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy（罗马A.Gemelli大学医院研究院）；Università Cattolica del Sacro Cuore, Rome, Italy（罗马圣心大学）；University Hospital of Strasbourg, France（斯特拉斯堡大学医院）
Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection
Center of Smart Health, School of Nursing, The Hong Kong Polytechnic University（智能健康中心、护理学院、香港理工大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Institute of Medical Intelligence and XR（医学智能与XR研究所）；The Chinese University of Hong Kong（香港中文大学）；Division of Hepatobiliopancreatic Surgery, Department of General Surgery（肝胆胰外科 division, 外科部）
Monocular Marker-free Patient-to-Image Intraoperative Registration for Cochlear Implant Surgery
Dept. of Computer Science, Vanderbilt University（计算机科学系，范德比尔特大学）；Dept. of Electrical and Computer Engineering, Vanderbilt University（电气与计算机工程系，范德比尔特大学）
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
Purpose: The objective of this investigation is to provide a comprehensive analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room. Methods: Using a data set of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluate feature based methods previously developed for surgical skill assessment mostly under benchtop settings. In addition, we present and validate two deep learning methods that directly assess skill using RGB videos. In the first method, we predict instrument tips as keypoints, and learn surgical skill using temporal…
F2PASeg: Feature Fusion for Pituitary Anatomy Segmentation in Endoscopic Surgery
1Centre for Artificial Intelligence；Robotics, Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences 2The University of Hong Kong 3Peking Union Medical College Hospital, Chinese Academy of Medical Science
Rochester Institute of Technology（罗切斯特理工学院）；University of Washington（华盛顿大学）；Seattle Children’s Hospital（西雅图儿童医院）
Abstract. Purpose: This paper presents a scheme for generating virtual intraoperative CT scans in order to improve surgical completeness in Endoscopic Sinus Surgeries (ESS). Approach: The work presents three methods, the tip motion-based, the tip trajectory-based, and the instrument based, along with non-parametric smoothing and Gaussian Process Regression, for virtual intraoperative CT generation. Results: The proposed methods studied and compared on ESS performed on cadavers. Surgical results show all three methods improve the Dice Similarity Coefficients > 86%, with F-score > 92% and precis…
CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家医学研究院，ICube，UMR7357，法国）；IHU Strasbourg, France（斯特拉斯堡IHU，法国）；Technical University of Munich, Germany（慕尼黑技术大学，德国）
Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis
Applied Robotics and AI Solutions (ARAS)（应用机器人与人工智能解决方案）；Faculties of Electrical and Computer Engineering（电气与计算机工程学院）；K.N. Toosi University of Technology（卡里姆·纳尼·托西技术大学）；Translational Ophthalmology Research Center（转化眼科研究中心）；Farabi Eye Hospital（法拉比眼科医院）；Tehran University of Medical Sciences（德黑兰医科大学）；Noor Ophthalmology Research Center（努尔眼科研究中心）；University of Alberta（阿尔伯塔大学）；Departments of Electrical and Computer Engineering & Biomedical Engineering（电气与计算机工程及生物医学工程系）
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；Auris Health, Inc.（奥里斯健康公司）；Johns Hopkins Medicine（约翰斯·霍普金斯医学院）；Nara Institute of Science and Technology（奈良科学技术研究所）；University of Texas（德克萨斯大学）；Texas Orthopedics（德克萨斯骨科医院）
Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. We propose a method for fully automatic registration using annotations produced by a neural network. Neural networks are trained to simultaneously segment anatomy and ide…
Revisiting the Evaluation Bias Introduced by Frame Sampling Strategies in Surgical Video Segmentation Using SAM2
1 Center for Biosystems；Biotech Data Science, Ghent University Global Campus, Incheon, Republic of Korea 2 IDLab, ELIS, Ghent University, Ghent, Belgium 3 Department of GI Surgery, Ghent University Hospital, Ghent, Belgium 4 Department of Human Structure；Repair, Ghent University, Ghent, Belgium 5 Department of HPB Surgery \& Liver Transplantation, Ghent University Hospital, Ghent, Belgium 6 Department of Mathematics, Computer Science；Statistics,\ University, Ghent, Belgium ( )
Balgrist University Hospital（巴尔格里斯特大学医院）；ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）
In surgical navigation, finding correspondence between preoperative plan and intraoperative anatomy, the so-called registration task, is imperative. One promising approach is to intraoperatively digitize anatomy and register it with the preoperative plan. State-of-the-art commercial navigation systems implement such approaches for pedicle screw placement in spinal fusion surgery. Although these systems improve surgical accuracy, they are not gold standard in clinical practice. Besides economical reasons, this may be due to their difficult integration into clinical workflows and unintuitive nav…
Bridging the Markerless Tracking Validation Gap: Intraoperative X-ray Workflow Integration
The Hamlyn Centre for Robotic Surgery, Imperial College London（伦敦帝国理工学院哈姆林外科机器人中心）；Imperial College Healthcare NHS Trust（帝国理工学院医疗保健 NHS 信托基金）；Stanford University School of Medicine（斯坦福大学医学院）
Biomedical Robotics Lab, Advanced Robotics（生物医学机器人实验室，先进机器人技术）；Istituto Italiano di Tecnologia（意大利技术研究所）；Department of Computer Science, Bioengineering, Robotics and Systems Engineering (DIBRIS)（计算机科学、生物工程、机器人与系统工程系（DIBRIS））；University of Genoa（热那亚大学）；Technical University of Munich（慕尼黑技术大学）；Harvard Medical School, Brigham and Women’s Hospital（哈佛医学院，布里洛妇女医院）
FOCUS: A Subspecialty Foundation Model for Intraoperative Central Nervous System Pathology
Beijing Neurosurgical Institute, Capital Medical University（首都医科大学附属北京神经外科研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；The First Hospital of Hebei Medical University（河北医科大学第一医院）
From Pre- to Intra-operative MRI: Predicting Brain Shift in Temporal Lobe Resection for Epilepsy Surgery
Surgical and Interventional Engineering（外科与介入工程）；King’s College London（伦敦国王学院）；National Hospital for Neurology and Neurosurgery（神经外科医院）；NHS（国家健康服务）；Neurosurgery, King’s College Hospital（神经外科，伦敦国王学院医院）；Fondazione IRCCS Ca’ Granda（IRCCS Ca’ Granda基金会）；Ospedale Maggiore di Milano（米兰大医院）
Beyond Role-Based Surgical Domain Modeling: Generalizable Re-Identification in the Operating Room
Chair for Computer Aided Medical Procedures, Technical University of Munich（计算机辅助医疗程序研究所，慕尼黑技术大学）；UCL Hawkes Institute, Dept. Computer Science, University College London（UCL哈维斯研究所，伦敦大学学院计算机科学系）；Department of General, Visceral, and Transplant Surgery, University Hospital, Ludwig-Maximilians-University（普通外科、 visceral外科及移植外科部门，路德维希-马克西米利安大学医院）
A Data-Centric Framework for Intraoperative Fluorescence Lifetime Imaging for Glioma Surgical Guidance
University of California, Davis（加州大学戴维斯分校）；Department of Biomedical Engineering（生物医学工程系）；Department of Pathology and Laboratory Medicine（病理学与实验室医学系）；Department of Neurological Surgery（神经外科系）
Peking University（北京大学）；Peking University Cancer Hospital（北京大学肿瘤医院）；Deepwise AI Lab（深睿医疗人工智能实验室）
Surgical skills have a great influence on surgical safety and patients' well-being. Traditional assessment of surgical skills involves strenuous manual efforts, which lacks efficiency and repeatability. Therefore, we attempt to automatically predict how well the surgery is performed using the surgical video. In this paper, a unified multi-path framework for automatic surgical skill assessment is proposed, which takes care of multiple composing aspects of surgical skills, including surgical tool usage, intraoperative event pattern, and other skill proxies. The dependency relationships among the…
SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding
University of Liverpool（利物浦大学）；City University of Hong Kong（香港城市大学）；University of Oxford（牛津大学）；University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Imperial College London（帝国理工学院）
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
Previous work on surgical skill assessment using intraoperative tool motion in the operating room (OR) has focused on highly-structured surgical tasks such as cholecystectomy. Further, these methods only considered generic motion metrics such as time and number of movements, which are of limited instructive value. In this paper, we developed and evaluated an automated approach to the surgical skill assessment of nasal septoplasty in the OR. The obstructed field of view and highly unstructured nature of septoplasty precludes trainees from efficiently learning the procedure. We propose a descrip…
Periacetabular osteotomy is a challenging surgical procedure for treating developmental hip dysplasia, providing greater coverage of the femoral head via relocation of a patient's acetabulum. Since fluoroscopic imaging is frequently used in the surgical workflow, computer-assisted X-Ray navigation of osteotomes and the relocated acetabular fragment should be feasible. We use intensity-based 2D/3D registration to estimate the pelvis pose with respect to fluoroscopic images, recover relative poses of multiple views, and triangulate landmarks which may be used for navigation. Existing similarity …
Towards Patient-Specific Deformable Registration in Laparoscopic Surgery
Biomedical Robotics Lab, Istituto Italiano di Tecnologia, Genoa, Italy（生物医学机器人实验室，意大利理工学院，热那亚）；Department of Computer Science, Bioengineering, Robotics and Systems Engineering (DIBRIS), University of Genoa, Italy（计算机科学、生物工程、机器人与系统工程系（DIBRIS），热那亚大学，意大利）；Harvard Medical School, Brigham and Women’s Hospital, Boston, USA（哈佛医学院，布里洛妇女医院，美国波士顿）
Hyperspectral imaging has the potential to improve intraoperative decision making if tissue characterisation is performed in real-time and with high-resolution. Hyperspectral snapshot mosaic sensors offer a promising approach due to their fast acquisition speed and compact size. However, a demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. Most state-of-the-art demosaicking algorithms require ground-truth training data with paired snapshot and high-resolution hyperspectral images, but such imagery pairs with the exact same scene are…
To ensure satisfactory clinical outcomes, surgical skill assessment must be objective, time-efficient, and preferentially automated - none of which is currently achievable. Video-based assessment (VBA) is being deployed in intraoperative and simulation settings to evaluate technical skill execution. However, VBA remains manually- and time-intensive and prone to subjective interpretation and poor inter-rater reliability. Herein, we propose a deep learning (DL) model that can automatically and objectively provide a high-stakes summative assessment of surgical skill execution based on video feeds…
GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation
King Abdullah University of Science and Technology (KAUST)（卡奥尔大学科学与技术学院）；Thapar Institute of Engineering and Technology（塔帕尔工程与技术学院）；The University of Queensland（昆士兰大学）；Gaash Research Lab, National Institute of Technology Srinagar（加什研究实验室，锡纳加尔国家理工学院）
Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心）；Centre For Medical Image Computing, University College London（伦敦大学学院医学图像计算中心）；Division of Surgery and Interventional Science, University College London（伦敦大学学院外科与介入科学系）
In this work we propose a method based on geometric deep learning to predict the complete surface of the liver, given a partial point cloud of the organ obtained during the surgical laparoscopic procedure. We introduce a new data augmentation technique that randomly perturbs shapes in their frequency domain to compensate the limited size of our dataset. The core of our method is a variational autoencoder (VAE) that is trained to learn a latent space for complete shapes of the liver. At inference time, the generative part of the model is embedded in an optimisation procedure where the latent re…
Real-Time Brain Tumor Detection in Intraoperative Ultrasound Using YOLO11: From Model Training to Deployment in the Operating Room
Department of Neurosurgery, Rio Hortega University Hospital（里奥霍尔特加大学医院神经外科部）；Department of Neurosurgery, Tata Memorial Centre, Homi Bhabha National Institute（塔塔纪念中心神经外科部，霍米·巴布哈国家研究所）；Biomedical Engineering Group, University of Valladolid（瓦拉多利德大学生物医学工程小组）；UMR 1253, iBrain, Université de Tours, Inserm（UMR 1253，iBrain， Tours大学，Inserm）；Department of Neurosurgery, CHRU de Tours（图尔CHRU医院神经外科部）；Department of Neurosrugery, ARNAS Civico Di Cristina Benfratelli Hospital（ARNAS公民迪克里斯塔弗拉蒂医院神经外科部）；Fondazione IRCCS Istituto Neurologico Carlo Besta（IRCCS神经学研究所卡尔罗·贝斯塔基金会）；University of Milan（米兰大学）；Johns Hopkins Medical School（约翰霍普金斯医学院）
Toward Reliable AR-Guided Surgical Navigation: Interactive Deformation Modeling with Data-Driven Biomechanics and Prompts
Department of Computer Science and Engineering, The Chinese University of Hong Kong, HKSAR, China（计算机科学与工程系，香港中文大学（深圳））；Center of Smart Health, School of Nursing, The Hong Kong Polytechnic University, HKSAR, China（智能健康中心，香港理工大学）；Department of Hepatobiliary Surgery, The Third Affiliated Hospital of Southern Medical University, Guangzhou, China（肝胆外科部，南方医科大学第三附属医院）
SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB)（电子、信息与生物工程系）；Politecnico di Milano（米兰理工大学）；IRCCS Humanitas Research Hospital（IRCCS人类itas研究医院）；UCL Hawkes Institute and Department of Computer Science（UCL Hawkes研究所和计算机科学系）；University College London（伦敦大学学院）；University of Manchester（曼彻斯特大学）
Generative data augmentation for biliary tract detection on intraoperative images
C.R.E.A.T.E. Consorzio di Ricerca per l’Energia, l’Automazione e le Tecnologie dell’Elettromagnetismo, Università degli Studi di Napoli Federico II（C.R.E.A.T.E.研究会，那不勒斯费德里科二世大学）；Department of Information Technology and Electrical Engineering, Università degli Studi di Napoli Federico II（信息科技与电气工程系，那不勒斯费德里科二世大学）；Department of Information Engineering, University of Padova（信息工程系，帕多瓦大学）；General Surgical Unit, Department of Public Health, Università degli Studi di Napoli Federico II（外科部，公共卫生系，那不勒斯费德里科二世大学）
