Purpose: This paper focuses on an automated analysis of surgical motion profiles for objective skill assessment and task recognition in robot-assisted surgery. Existing techniques heavily rely on conventional statistic measures or shallow modelings based on hand-engineered features and gesture segmentation. Such developments require significant expert knowledge, are prone to errors, and are less efficient in online adaptive training systems. Methods: In this work, we present an efficient analytic framework with a parallel deep learning architecture, SATR-DL, to assess trainee expertise and recognize surgical training activity. Through an end-to-end learning technique, abstract information of spatial representations and temporal dynamics is jointly obtained directly from raw motion sequences. Results: By leveraging a shared high-level representation learning, the resulting model is successful in the recognition of trainee skills and surgical tasks, suturing, needle-passing, and knot-tying. Meanwhile, we explore the use of ensemble in classification at the trial level, where the SATR-DL outperforms state-of-the-art performance by achieving accuracies of 0.960 and 1.000 in skill assessment and task recognition, respectively. Conclusion: This study highlights the potential of SATR-DL to provide improvements for an efficient data-driven assessment in intelligent robotic surgery.
Autonomy is the frontier of research in robotic surgery and its aim is to improve the quality of surgical procedures in the next future. One fundamental requirement for autonomy is advanced perception capability through vision sensors. In this paper, we propose a novel calibration technique for a surgical scenario with da Vinci robot. Calibration of the camera and the robot is necessary for precise positioning of the tools in order to emulate the high performance surgeons. Our calibration technique is tailored for RGB-D camera. Different tests performed on relevant use cases for surgery prove …
The use of robots in minimally invasive surgery has improved the quality of standard surgical procedures. So far, only the automation of simple surgical actions has been investigated by researchers, while the execution of structured tasks requiring reasoning on the environment and the choice among multiple actions is still managed by human surgeons. In this paper, we propose a framework to implement surgical task automation. The framework consists of a task-level reasoning module based on answer set programming, a low-level motion planning module based on dynamic movement primitives, and a sit…
SurgBox: Agent-Driven Operating Room Sandbox with Surgery Copilot
CAIR, HKISI, CAS（中国科学院香港中文大学信息安全研究所智能计算研究中心）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；Peking Union Medical College Hospital（北京协和医院）
Intuitive Surgical Inc.（直觉外科公司）；California Institute of Technology（加州理工学院）
This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual…
MIT（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Olympus Corporation（奥林巴斯公司）
Comprehension of surgical workflow is the foundation upon which artificial intelligence (AI) and machine learning (ML) holds the potential to assist intraoperative decision-making and risk mitigation. In this work, we move beyond mere identification of past surgical phases, into the prediction of future surgical steps and specification of the transitions between them. We use a novel Generative Adversarial Network (GAN) formulation to sample future surgical phases trajectories conditioned on past video frames from laparoscopic cholecystectomy (LC) videos and compare it to state-of-the-art appro…
Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；University College London（伦敦大学学院）；National University of Singapore（新加坡国立大学）
Registering Neural 4D Gaussians for Endoscopic Surgery
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Shenzhen Research Institute, CUHK（香港中文大学深圳研究院）；KTH, Kungliga Tekniska högskolan Royal Institute of Technology（瑞典皇家理工学院）；Hong Kong Applied Science and Technology Research Institute Company Limited（香港应用科技研究院有限公司）
OSSAR: Towards Open-Set Surgical Activity Recognition in Robot-assisted Surgery
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Beijing Institute of Technology（北京理工大学）；Qilu Hospital of Shandong University（山东大学齐鲁医院）；Tongji University（同济大学）；University College London（伦敦大学学院）
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）
Objective: Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classif…
Asan Medical Center（峨山医疗中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Chungnam National University School of Medicine（忠南大学医学院）
Some endoscopic surgeries require a surgeon to hold the endoscope with one hand and the surgical instruments with the other hand to perform the actual surgery with correct vision. Recent technical advances in deep learning as well as in robotics can introduce robotics to these endoscopic surgeries. This can have numerous advantages by freeing one hand of the surgeon, which will allow the surgeon to use both hands and to use more intricate and sophisticated techniques. Recently, deep learning with convolutional neural network achieves state-of-the-art results in computer vision. Therefore, the …
Despite significant developments in the design of surgical robots and automated techniques for objective evaluation of surgical skills, there are still challenges in ensuring safety in robot-assisted minimally-invasive surgery (RMIS). This paper presents a runtime monitoring system for the detection of executional errors during surgical tasks through the analysis of kinematic data. The proposed system incorporates dual Siamese neural networks and knowledge of surgical context, including surgical tasks and gestures, their distributional similarities, and common error modes, to learn the differe…
Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Department of Thoracic Surgery, the Seventh Affiliated Hospital, Sun Yat-sen University（中山大学第七附属医院胸外科部门）；Bioengineering/Imperial-X, Imperial College London（生物工程/Imperial-X，帝国理工学院伦敦分校）；ROAS Thrust, Hong Kong University of Science and Technology (Guangzhou)（ROAS项目，香港科技大学（广州））；Department of Electronic and Computer Engineering, Hong Kong SAR（香港特别行政区电子与计算机工程系）
Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心）；National University of Singapore（新加坡国立大学）
University of Hyogo（兵库大学）；Takatsuki Hospital（高槻医院）；International University of Business Agriculture & Technology（国际商业农业与技术大学）
Total knee arthroplasty (TKA) is a commonly performed surgical procedure to mitigate knee pain and improve functions for people with knee arthritis. The procedure is complicated due to the different surgical tools used in the stages of surgery. The recognition of surgical tools in real-time can be a solution to simplify surgical procedures for the surgeon. Also, the presence and movement of tools in surgery are crucial information for the recognition of the operational phase and to identify the surgical workflow. Therefore, this research proposes the development of a real-time system for the r…
SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery
University of Michigan（密歇根大学）；University of Edinburgh（爱丁堡大学）；Vanderbilt University Medical Center（范德比尔特大学医学中心）；University of Colorado（科罗拉多大学）；University of Rochester（罗切斯特大学）；Michigan Medicine（密歇根医学）
Automatic recognition of surgical activities in the operating room (OR) is a key technology for creating next generation intelligent surgical devices and workflow monitoring/support systems. Such systems can potentially enhance efficiency in the OR, resulting in lower costs and improved care delivery to the patients. In this paper, we investigate automatic surgical activity recognition in robot-assisted operations. We collect the first large-scale dataset including 400 full-length multi-perspective videos from a variety of robotic surgery cases captured using Time-of-Flight cameras. We densely…
The University of Texas at Dallas（得克萨斯大学达拉斯分校）；UT Southwestern Medical Center（得克萨斯大学西南医学中心）
With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge. We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series…
California Institute of Technology（加州理工学院）；University of Southern California（南加州大学）
Surgery is a high-stakes domain where surgeons must navigate critical anatomical structures and actively avoid potential complications while achieving the main task at hand. Such surgical activity has been shown to affect long-term patient outcomes. To better understand this relationship, whose mechanics remain unknown for the majority of surgical procedures, we hypothesize that the core elements of surgery must first be quantified in a reliable, objective, and scalable manner. We believe this is a prerequisite for the provision of surgical feedback and modulation of surgeon performance in pur…
Imperial College London（帝国理工学院）；University of Cambridge（剑桥大学）；PAII Inc（PAII公司）
In Robot-Assisted Minimally Invasive Surgery (RAMIS), a camera assistant is normally required to control the position and zooming ratio of the laparoscope, following the surgeon's instructions. However, moving the laparoscope frequently may lead to unstable and suboptimal views, while the adjustment of zooming ratio may interrupt the workflow of the surgical operation. To this end, we propose a multi-scale Generative Adversarial Network (GAN)-based video super-resolution method to construct a framework for automatic zooming ratio adjustment. It can provide automatic real-time zooming for high-…
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院威康/EPSRC介入与外科科学中心）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
Semantic tool segmentation in surgical videos is important for surgical scene understanding and computer-assisted interventions as well as for the development of robotic automation. The problem is challenging because different illumination conditions, bleeding, smoke and occlusions can reduce algorithm robustness. At present labelled data for training deep learning models is still lacking for semantic surgical instrument segmentation and in this paper we show that it may be possible to use robot kinematic data coupled with laparoscopic images to alleviate the labelling problem. We propose a ne…
VS-Assistant: Versatile Surgery Assistant on the Demand of Surgeons
Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（香港科学院人工智能与机器人中心（CAIR））；The Chinese University of Hong Kong（香港中文大学）；King’s College London（伦敦国王学院）
Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgery
The Chinese University of Hong Kong（香港中文大学）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）
Inserm, UMR 1101（法国国家健康与医学研究院）；Univ Bretagne Occidentale（西布列塔尼大学）；INSTITUT TELECOM（法国电信学院；布雷斯特电信学院）；TELECOM Bretagne（布雷斯特大学医院）；CHRU Brest
The amount of surgical data, recorded during video-monitored surgeries, has extremely increased. This paper aims at improving existing solutions for the automated analysis of cataract surgeries in real time. Through the analysis of a video recording the operating table, it is possible to know which instruments exit or enter the operating table, and therefore which ones are likely being used by the surgeon. Combining these observations with observations from the microscope video should enhance the overall performance of the systems. To this end, the proposed solution is divided into two main pa…
National University of Singapore（新加坡国立大学）；Imperial College London（伦敦帝国学院）；Singapore General Hospital（新加坡中央医院）
Learning to infer graph representations and performing spatial reasoning in a complex surgical environment can play a vital role in surgical scene understanding in robotic surgery. For this purpose, we develop an approach to generate the scene graph and predict surgical interactions between instruments and surgical region of interest (ROI) during robot-assisted surgery. We design an attention link function and integrate with a graph parsing network to recognize the surgical interactions. To embed each node with corresponding neighbouring node features, we further incorporate SageConv into the …
Surgical-MambaLLM: Mamba2-enhanced Multimodal Large Language Model for VQLA in Robotic Surgery
Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Imperial College London（帝国理工学院伦敦分校）；The Hong Kong University of Science and Technology（香港科学与技术大学）
Imperial College London（伦敦帝国理工学院）；Tsinghua University（清华大学）；NYU Abu Dhabi（纽约大学阿布扎比分校）；Wuhan University（武汉大学）；University College London（伦敦大学学院）；University of Liverpool（利物浦大学）
Large-scale Self-supervised Video Foundation Model for Intelligent Surgery
The Hong Kong University of Science and Technology（香港科技大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；The Hong Kong Polytechnic University（香港理工大学）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）；The Chinese University of Hong Kong（香港中文大学）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Shenzhen-Hong Kong Collaborative Innovation Research Institute（深港协同创新研究院）
CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery
Chair for Computer-Aided Medical Procedures, Technical University Munich, Germany（计算机辅助医疗程序研究所，慕尼黑技术大学，德国）；Carl Zeiss AG, Munich, Germany（蔡司股份公司，慕尼黑，德国）
OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding
Monash University（莫纳什大学）；Airdoc-Monash Research（Airdoc-莫纳什研究所）；Airdoc；Bosch Corporate Research（博世企业研究院）；Cornell University（康奈尔大学）；King’s College London（伦敦国王学院）；Institute for Infocomm Research, A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））；City University of Macau（澳门城市大学）；Eye Hospital, Wenzhou Medical University（温州医科大学附属眼视光医院）
Rochester Institute of Technology（罗切斯特理工学院）；University of Washington（华盛顿大学）；Seattle Children’s Hospital（西雅图儿童医院）
Abstract. Purpose: This paper presents a scheme for generating virtual intraoperative CT scans in order to improve surgical completeness in Endoscopic Sinus Surgeries (ESS). Approach: The work presents three methods, the tip motion-based, the tip trajectory-based, and the instrument based, along with non-parametric smoothing and Gaussian Process Regression, for virtual intraoperative CT generation. Results: The proposed methods studied and compared on ESS performed on cadavers. Surgical results show all three methods improve the Dice Similarity Coefficients > 86%, with F-score > 92% and precis…
Surgical-DeSAM: Decoupling SAM for Instrument Segmentation in Robotic Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（Wellcome/EPSRC介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；Dept. of Computer Science, University College London（伦敦大学学院计算机科学系）
Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)
Regensburg Medical Image Computing (ReMIC), OTH Regensburg（雷根斯堡医学影像计算中心（ReMIC））；Regensburg Center of Biomedical Engineering (RCBE), OTH Regensburg（雷根斯堡生物医学工程中心（RCBE））；Regensburg University（雷根斯堡大学）；Regensburg Center of Health Sciences（雷根斯堡健康科学中心）；Technical University of Munich (TUM)（慕尼黑技术大学）；TUM University Hospital（慕尼黑技术大学医院）；Biomedical Image Analysis Group, Department of Computing, Imperial College London（生物医学影像分析组，计算部门，伦敦帝国学院）；Research Group MITI, TUM University Hospital, School of Medicine（MITI研究组，慕尼黑技术大学医院，医学学院）；Department of Surgery, TUM University Hospital, School of Medicine（外科部门，慕尼黑技术大学医院，医学学院）
EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd.（华为技术有限公司中央研究院理论实验室2012实验室）；Chair of Computer Aided Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助程序教席）；Department of Gastroenterology, Qilu Hospital of Shandong University（山东大学齐鲁医院消化内科）
Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；Universiti Malaya（马来亚大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港智能系统研究所人工智能与机器人中心）；University College London（伦敦大学学院）
Prior work demonstrated the ability of machine learning to automatically recognize surgical workflow steps from videos. However, these studies focused on only a single type of procedure. In this work, we analyze, for the first time, surgical step recognition on four different laparoscopic surgeries: Cholecystectomy, Right Hemicolectomy, Sleeve Gastrectomy, and Appendectomy. Inspired by the traditional apprenticeship model, in which surgical training is based on the Halstedian method, we paraphrase the "see one, do one, teach one" approach for the surgical intelligence domain as "train one, cla…
