Minimally invasive surgery is highly operator dependant with a lengthy procedural time causing fatigue to surgeon and risks to patients such as injury to organs, infection, bleeding, and complications of anesthesia. To mitigate such risks, real-time systems are desired to be developed that can provide intra-operative guidance to surgeons. For example, an automated system for tool localization, tool (or tissue) tracking, and depth estimation can enable a clear understanding of surgical scenes preventing miscalculations during surgical procedures. In this work, we present a systematic review of recent machine learning-based approaches including surgical tool localization, segmentation, tracking, and 3D scene perception. Furthermore, we provide a detailed overview of publicly available benchmark datasets widely used for surgical navigation tasks. While recent deep learning architectures have shown promising results, there are still several open research problems such as a lack of annotated datasets, the presence of artifacts in surgical scenes, and non-textured surfaces that hinder 3D reconstruction of the anatomical structures. Based on our comprehensive review, we present a discussion on current gaps and needed steps to improve the adaptation of technology in surgery.
SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splatting
The Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学）；Shenzhen Research Institute, CUHK, Shenzhen, China（深圳研究学院）；Technical University of Munich, Munich, Germany（慕尼黑技术大学）；University of Strasbourg & IHU Strasbourg, Strasbourg, France（斯特拉斯堡大学及斯特拉斯堡IHU）；University College London, London, United Kingdom（伦敦大学学院）
Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes
ROCS（罗克辛研究所）；Balgrist University Hospital（巴尔格里斯大学医院）；University of Zurich（苏黎世大学）；Dept. of Orthopedics（骨科部门）；Computer Vision and Geometry Group（计算机视觉与几何组）；ETH Zurich（苏黎世联邦理工学院）
The Dresden Dataset for 4D Reconstruction of Non-Rigid Abdominal Surgical Scenes
Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, and Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany（转化外科肿瘤中心部门，肿瘤疾病国家中心（NCT），NCT/UCC德累斯顿，由DKFZ、医学院和卡尔·戈斯瓦尔德·卡尔医院、德累斯顿技术大学以及德累斯顿-罗斯恩多夫赫尔姆霍兹中心（HZDR）组成的合作机构，德累斯顿，德国）；Department of Translational Surgical Oncology, NCT/UCC Dresden, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（转化外科肿瘤中心部门，NCT/UCC德累斯顿，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；Department for Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden（visceral、胸腔和血管外科部门，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technical University Dresden（人机交互触觉互联网中心（CeTI），德累斯顿技术大学）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（计算机科学与工程部门，香港中文大学）
The University of Sydney（悉尼大学）；University of Technology Sydney（悉尼科技大学）
Registration is a transformation estimation problem between two point clouds, which has a unique and critical role in numerous computer vision applications. The developments of optimization-based methods and deep learning methods have improved registration robustness and efficiency. Recently, the combinations of optimization-based and deep learning methods have further improved performance. However, the connections between optimization-based and deep learning methods are still unclear. Moreover, with the recent development of 3D sensors and 3D reconstruction techniques, a new research directio…
Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review
MBZ University of AI（MBZ人工智能大学）；Hamad Bin Khalifa University（哈马德·本·卡西姆大学）；Birmingham City University（伯明翰城市大学）；Qatar University（卡塔尔大学）；University Hospitals Birmingham and University of Birmingham（伯明翰大学医院和伯明翰大学）
SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；University of Oxford（牛津大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；Shanghai Jiao Tong University（上海交通大学）；Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）
MIT（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Olympus Corporation（奥林巴斯公司）
Comprehension of surgical workflow is the foundation upon which artificial intelligence (AI) and machine learning (ML) holds the potential to assist intraoperative decision-making and risk mitigation. In this work, we move beyond mere identification of past surgical phases, into the prediction of future surgical steps and specification of the transitions between them. We use a novel Generative Adversarial Network (GAN) formulation to sample future surgical phases trajectories conditioned on past video frames from laparoscopic cholecystectomy (LC) videos and compare it to state-of-the-art appro…
Systematic Evaluation and Guidelines for Segment Anything Model in Surgical Video Analysis
Global College（全球学院）；Shanghai Jiao Tong University（上海交通大学）；Department of Automation（自动化系）；National University of Singapore（新加坡国立大学）；Department of Computer Science and Engineering（计算机科学与工程系）
OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding
Monash University（莫纳什大学）；Airdoc-Monash Research（Airdoc-莫纳什研究所）；Airdoc；Bosch Corporate Research（博世企业研究院）；Cornell University（康奈尔大学）；King’s College London（伦敦国王学院）；Institute for Infocomm Research, A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））；City University of Macau（澳门城市大学）；Eye Hospital, Wenzhou Medical University（温州医科大学附属眼视光医院）
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
Purpose: The objective of this investigation is to provide a comprehensive analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room. Methods: Using a data set of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluate feature based methods previously developed for surgical skill assessment mostly under benchtop settings. In addition, we present and validate two deep learning methods that directly assess skill using RGB videos. In the first method, we predict instrument tips as keypoints, and learn surgical skill using temporal…
University of Strasbourg（斯特拉斯堡大学）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅里基金会天主教大学综合医院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Basel（巴塞尔大学）
Imperial College London（伦敦帝国理工学院）；Tsinghua University（清华大学）；NYU Abu Dhabi（纽约大学阿布扎比分校）；Wuhan University（武汉大学）；University College London（伦敦大学学院）；University of Liverpool（利物浦大学）
Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensive surgical perception framework, Semantic-SuPer, that integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes, benefiting dow…
Large-scale Self-supervised Video Foundation Model for Intelligent Surgery
The Hong Kong University of Science and Technology（香港科技大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；The Hong Kong Polytechnic University（香港理工大学）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）；The Chinese University of Hong Kong（香港中文大学）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Shenzhen-Hong Kong Collaborative Innovation Research Institute（深港协同创新研究院）
EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery
School of Computer Science & Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science, University of Nottingham Ningbo China（诺丁汉大学宁波校区计算机科学学院）；Wenzhou Medical University（温州医科大学）；School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）
SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
Samsung Research（三星研究所）；Center for Humanoid Research, Korea Institute of Science and Technology（人类研究学院，韩国科学技术院）；Department of plastic surgery, College of medicine, Korea University（医学院整形外科部，韩国大学）；Department of orthopedic surgery, College of medicine, Korea University（医学院骨科部，韩国大学）
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；University of Heidelberg（海德堡大学）；National Center for Tumor Diseases, Partner Site Dresden（国家肿瘤疾病中心德累斯顿分中心）；HIDSS4Health – Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；University of Calabria（卡拉布里亚大学）；understandAI GmbH（understandAI 有限公司）；University of Tübingen（蒂宾根大学）；BMW Group（宝马集团）；KARL STORZ SE & Co. KG（卡尔史托斯公司）；TU Dresden（德累斯顿工业大学）
Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a speci…
SurgVisAgent: Multimodal Agentic Model for Versatile Surgical Visual Enhancement
University of Chinese Academy of Sciences Multimedia Department, Xiaomi Inc Chinese Academy of Sciences, Institute of Automation Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Raytheon BBN Technologies（雷神BBN科技公司）；Army Research Laboratory（陆军研究实验室）
Surgical training in medical school residency programs has followed the apprenticeship model. The learning and assessment process is inherently subjective and time-consuming. Thus, there is a need for objective methods to assess surgical skills. Here, we use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to systematically survey the literature on the use of Deep Neural Networks for automated and objective surgical skill assessment, with a focus on kinematic data as putative markers of surgical competency. There is considerable recent interest in deep…
Ophora: A Large-Scale Data-Driven Text-Guided Ophthalmic Surgical Video Generation Model
Shanghai Artificial Intelligence Laboratory, China（上海人工智能实验室）；Shanghai Jiao Tong University, China（上海交通大学）；Monash University, Australia（墨尔本大学）；East China Normal University, China（华东师范大学）；Eye Hospital, Wenzhou Medical University, China（温州医学院眼医院）；Shanghai Academy of Artificial Intelligence for Science, China（上海人工智能科学研究院）；Imperial College London, UK（伦敦帝国理工学院）；Shanghai Innovation Institute, China（上海创新研究院）
The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark
ICube, University of Strasbourg, CNRS（ICube，斯特拉斯堡大学，CNRS）；IHU Strasbourg（斯特拉斯堡IHU）；Institute for Research against Digestive Cancer (IRCAD)（消化道癌症研究所）；Fondazione Policlinico Universitario A. Gemelli IRCCS（A.Gemelli大学附属医院）
Politecnico di Milano（米兰理工大学）；Istituto Italiano di Tecnologia（意大利技术研究院）；German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Heidelberg University Hospital（海德堡大学医院）
Objective: Surgical data science is evolving into a research field that aims to observe everything occurring within and around the treatment process to provide situation-aware data-driven assistance. In the context of endoscopic video analysis, the accurate classification of organs in the field of view of the camera proffers a technical challenge. Herein, we propose a new approach to anatomical structure classification and image tagging that features an intrinsic measure of confidence to estimate its own performance with high reliability and which can be applied to both RGB and multispectral i…
SurgBox: Agent-Driven Operating Room Sandbox with Surgery Copilot
CAIR, HKISI, CAS（中国科学院香港中文大学信息安全研究所智能计算研究中心）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；Peking Union Medical College Hospital（北京协和医院）
CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery
Chair for Computer-Aided Medical Procedures, Technical University Munich, Germany（计算机辅助医疗程序研究所，慕尼黑技术大学，德国）；Carl Zeiss AG, Munich, Germany（蔡司股份公司，慕尼黑，德国）
Surgeons vs. Computer Vision: A comparative analysis on surgical phase recognition capabilities
Orsi Academy（奥西学院）；Faculty of Engineering Science（工程科学学院）；KU Leuven（鲁文大学）；Department of Urology（泌尿科部门）；OLV Aalst Hospital（阿勒斯医院）；School of BMEIS（生物医学工程与科学学院）
DARIL: When Imitation Learning outperforms Reinforcement Learning in Surgical Action Planning
Surgical and Interventional Engineering, King's College London, London, UK（外科与介入工程，伦敦国王学院，伦敦，英国）；Interventional Engineering, King's College London, London, UK（介入工程，伦敦国王学院，伦敦，英国）
该研究在 CholecT50 上比较手术动作规划中的 IL 与 RL，提出 DARIL 基线，发现其显著优于三种 RL 方法，并揭示专家测试分布对 IL 的系统性偏好。
Expert Consensus on Criteria for the Automated Assessment of Laparoscopic Camera Navigation
University Medical Center Göttingen（哥廷根大学医学中心）；Fraunhofer Research Institution for Individualized and Cell-based Medical Engineering (IMTE)（弗劳恩霍夫个性化与细胞医学工程研究所）
