Purpose: Basic surgical skills of suturing and knot tying are an essential part of medical training. Having an automated system for surgical skills assessment could help save experts time and improve training efficiency. There have been some recent attempts at automated surgical skills assessment using either video analysis or acceleration data. In this paper, we present a novel approach for automated assessment of OSATS based surgical skills and provide an analysis of different features on multi-modal data (video and accelerometer data). Methods: We conduct the largest study, to the best of our knowledge, for basic surgical skills assessment on a dataset that contained video and accelerometer data for suturing and knot-tying tasks. We introduce "entropy based" features - Approximate Entropy (ApEn) and Cross-Approximate Entropy (XApEn), which quantify the amount of predictability and regularity of fluctuations in time-series data. The proposed features are compared to existing methods of Sequential Motion Texture (SMT), Discrete Cosine Transform (DCT) and Discrete Fourier Transform (DFT), for surgical skills assessment. Results: We report average performance of different features across all applicable OSATS criteria for suturing and knot tying tasks. Our analysis shows that the proposed entropy based features out-perform previous state-of-the-art methods using video data. For accelerometer data, our method performs better for suturing only. We also show that fusion of video and acceleration features can improve overall performance with the proposed entropy features achieving highest accuracy. Conclusions: Automated surgical skills assessment can be achieved with high accuracy using the proposed entropy features. Such a system can significantly improve the efficiency of surgical training in medical schools and teaching hospitals.
Peking University（北京大学）；Peking University Cancer Hospital（北京大学肿瘤医院）；Deepwise AI Lab（深睿医疗人工智能实验室）
Surgical skills have a great influence on surgical safety and patients' well-being. Traditional assessment of surgical skills involves strenuous manual efforts, which lacks efficiency and repeatability. Therefore, we attempt to automatically predict how well the surgery is performed using the surgical video. In this paper, a unified multi-path framework for automatic surgical skill assessment is proposed, which takes care of multiple composing aspects of surgical skills, including surgical tool usage, intraoperative event pattern, and other skill proxies. The dependency relationships among the…
Peking University（北京大学）；Peng Cheng Lab（鹏城实验室）；Deepwise AI Lab（深睿医疗人工智能实验室）；Peking University Cancer Hospital（北京大学肿瘤医院）
Surgical skill assessment is important for surgery training and quality control. Prior works on this task largely focus on basic surgical tasks such as suturing and knot tying performed in simulation settings. In contrast, surgical skill assessment is studied in this paper on a real clinical dataset, which consists of fifty-seven in-vivo laparoscopic surgeries and corresponding skill scores annotated by six surgeons. From analyses on this dataset, the clearness of operating field (COF) is identified as a good proxy for overall surgical skills, given its strong correlation with overall skills a…
IRIMAS（IRIMAS研究所）；Université de Haute-Alsace（上阿尔萨斯大学）
The need for automatic surgical skills assessment is increasing, especially because manual feedback from senior surgeons observing junior surgeons is prone to subjectivity and time consuming. Thus, automating surgical skills evaluation is a very important step towards improving surgical practice. In this paper, we designed a Convolutional Neural Network (CNN) to evaluate surgeon skills by extracting patterns in the surgeon motions performed in robotic surgery. The proposed method is validated on the JIGSAWS dataset and achieved very competitive results with 100% accuracy on the suturing and ne…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
Purpose: The objective of this investigation is to provide a comprehensive analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room. Methods: Using a data set of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluate feature based methods previously developed for surgical skill assessment mostly under benchtop settings. In addition, we present and validate two deep learning methods that directly assess skill using RGB videos. In the first method, we predict instrument tips as keypoints, and learn surgical skill using temporal…
Five billion people in the world lack access to quality surgical care. Surgeon skill varies dramatically, and many surgical patients suffer complications and avoidable harm. Improving surgical training and feedback would help to reduce the rate of complications, half of which have been shown to be preventable. To do this, it is essential to assess operative skill, a process that currently requires experts and is manual, time consuming, and subjective. In this work, we introduce an approach to automatically assess surgeon performance by tracking and analyzing tool movements in surgical videos, …
Johnson & Johnson（强生公司）；Johnson & Johnson Medical GmbH（强生医疗有限公司）
Mastering the technical skills required to perform surgery is an extremely challenging task. Video-based assessment allows surgeons to receive feedback on their technical skills to facilitate learning and development. Currently, this feedback comes primarily from manual video review, which is time-intensive and limits the feasibility of tracking a surgeon's progress over many cases. In this work, we introduce a motion-based approach to automatically assess surgical skills from surgical case video feed. The proposed pipeline first tracks surgical tools reliably to create motion trajectories and…
Intuitive, Inc.（直觉外科公司）；Muroran Institute of Technology（室兰工业大学）；University College London（伦敦大学学院）；Shandong University（山东大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）
Timely and effective feedback within surgical training plays a critical role in developing the skills required to perform safe and efficient surgery. Feedback from expert surgeons, while especially valuable in this regard, is challenging to acquire due to their typically busy schedules, and may be subject to biases. Formal assessment procedures like OSATS and GEARS attempt to provide objective measures of skill, but remain time-consuming. With advances in machine learning there is an opportunity for fast and objective automated feedback on technical skills. The SimSurgSkill 2021 challenge (hos…
National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯医学院暨大学医院）
Purpose: A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill. Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability …
The University of Tokyo（东京大学）；RIKEN（理化学研究所）；Dalian University of Technology（大连理工大学）；AIST（产业技术综合研究所）
Automated video-based assessment of surgical skills is a promising task in assisting young surgical trainees, especially in poor-resource areas. Existing works often resort to a CNN-LSTM joint framework that models long-term relationships by LSTMs on spatially pooled short-term CNN features. However, this practice would inevitably neglect the difference among semantic concepts such as tools, tissues, and background in the spatial dimension, impeding the subsequent temporal relationship modeling. In this paper, we propose a novel skill assessment framework, Video Semantic Aggregation (ViSA), wh…
College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）
Purpose: Manual feedback in basic RMIS training can consume a significant amount of time from expert surgeons' schedule and is prone to subjectivity. While VR-based training tasks can generate automated score reports, there is no mechanism of generating automated feedback for surgeons performing basic surgical tasks in RMIS training. In this paper, we explore the usage of different holistic features for automated skill assessment using only robot kinematic data and propose a weighted feature fusion technique for improving score prediction performance. Methods: We perform our experiments on the…
University of Zagreb（萨格勒布大学）；University of Twente（特文特大学）；University of Groningen（格罗宁根大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Canisius-Wilhelmina Hospital（卡尼修斯-威廉明娜医院）
Stanford University（斯坦福大学）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）
Open procedures represent the dominant form of surgery worldwide. Artificial intelligence (AI) has the potential to optimize surgical practice and improve patient outcomes, but efforts have focused primarily on minimally invasive techniques. Our work overcomes existing data limitations for training AI models by curating, from YouTube, the largest dataset of open surgical videos to date: 1997 videos from 23 surgical procedures uploaded from 50 countries. Using this dataset, we developed a multi-task AI model capable of real-time understanding of surgical behaviors, hands, and tools - the buildi…
CoRe-DA: Contrastive Regression for Unsupervised Domain Adaptation in Surgical Skill Assessment
UCL Hawkes Institute, University College London（伦敦大学学院霍克斯研究所）；Dept of Computer Science, University College London（伦敦大学学院计算机科学系）；The Griffin Institute（格里芬研究所）
To ensure satisfactory clinical outcomes, surgical skill assessment must be objective, time-efficient, and preferentially automated - none of which is currently achievable. Video-based assessment (VBA) is being deployed in intraoperative and simulation settings to evaluate technical skill execution. However, VBA remains manually- and time-intensive and prone to subjective interpretation and poor inter-rater reliability. Herein, we propose a deep learning (DL) model that can automatically and objectively provide a high-stakes summative assessment of surgical skill execution based on video feeds…
Heidelberg University Hospital（海德堡大学医院）；National Center for Tumor Diseases (NCT) Heidelberg（海德堡国家肿瘤疾病中心）；KARL STORZ SE & Co. KG（卡尔史托斯股份有限公司）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center dataset. In this work we investigated the generalizability of phase recognition algorithms in a multi-center setting including more difficult recognit…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Raytheon BBN Technologies（雷神BBN科技公司）；Army Research Laboratory（陆军研究实验室）
Surgical training in medical school residency programs has followed the apprenticeship model. The learning and assessment process is inherently subjective and time-consuming. Thus, there is a need for objective methods to assess surgical skills. Here, we use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to systematically survey the literature on the use of Deep Neural Networks for automated and objective surgical skill assessment, with a focus on kinematic data as putative markers of surgical competency. There is considerable recent interest in deep…
Purpose: This paper focuses on an automated analysis of surgical motion profiles for objective skill assessment and task recognition in robot-assisted surgery. Existing techniques heavily rely on conventional statistic measures or shallow modelings based on hand-engineered features and gesture segmentation. Such developments require significant expert knowledge, are prone to errors, and are less efficient in online adaptive training systems. Methods: In this work, we present an efficient analytic framework with a parallel deep learning architecture, SATR-DL, to assess trainee expertise and rec…
Indian Institute of Technology Delhi（印度理工学院德里分校）；All India Institute of Medical Sciences（全印度医学科学研究所）
Neuro-endoscopy is a challenging minimally invasive neurosurgery that requires surgical skills to be acquired using training methods different from the existing apprenticeship model. There are various training systems developed for imparting fundamental technical skills in laparoscopy where as limited systems for neuro-endoscopy. Neuro-Endo-Trainer was a box-trainer developed for endo-nasal transsphenoidal surgical skills training with video based offline evaluation system. The objective of the current study was to develop a modified version (Neuro-Endo-Trainer-Online Assessment System (NET-OA…
Looking Beyond the Scale: Do Surgical Skill Models Learn Transferable Representations Across Assessment Rubrics?
National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；DKFZ（德国癌症研究中心）；Faculty of Medicine and University Hospital Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯医学院及大学医院）；TUD Dresden University of Technology（德累斯顿工业大学）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；BMFTR Research Hub 6G-Life（BMFTR 6G生活研究中心）；Deutsches Krebsforschungszentrum (DKFZ)（德国癌症研究中心）；The Centre for Tactile Internet with Human-in-the-Loop (CeTI)（人在回路触觉互联网中心）
The University of Texas at Dallas（得克萨斯大学达拉斯分校）；UT Southwestern Medical Center（得克萨斯大学西南医学中心）
With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge. We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series…
Explainable AI-Powered Framework for Video-Based Skill Assessment in Cataract Surgery
K.N. Toosi University of Technology（霍加托西大学）；Faculties of Electrical and Computer Engineering（电气与计算机工程学院）；Applied Robotics and AI Solutions (ARAS)（应用机器人与人工智能解决方案（ARAS））
Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment
UCL Hawkes Institute, University College London, UK Dept of Medical Physics \& Biomedical Engineering, University College London, UK Medtronic, Digital Technologies, UK Gloucestershire Hospitals NHS Foundation Trust, UK Dept of Computer Science, University College London, UK University College London Hospitals NHS Foundation Trust, UK Royal Free Hospital NHS Foundation Trust, UK The Griffin Institute, UK
A benchmark for video-based laparoscopic skill analysis and assessment
National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；University Hospital Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯大学医院）；TUD Dresden University of Technology（德累斯顿工业大学）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；The Centre for Tactile Internet with Human-in-the-Loop (CeTI)（人在环触觉互联网中心）；Medical Faculty and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学医学院暨卡尔·古斯塔夫·卡鲁斯大学医院）；Municipal Hospital Dresden-Friedrichstadt（德累斯顿-弗里德里希施塔特市立医院）
National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；University Hospital Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯大学医院）；TU Dresden（德累斯顿工业大学）；Centre for Tactile Internet with Human-in-the-Loop (CeTI)（人在环触觉互联网中心）
Automatically recognizing surgical gestures is a crucial step towards a thorough understanding of surgical skill. Possible areas of application include automatic skill assessment, intra-operative monitoring of critical surgical steps, and semi-automation of surgical tasks. Solutions that rely only on the laparoscopic video and do not require additional sensor hardware are especially attractive as they can be implemented at low cost in many scenarios. However, surgical gesture recognition based only on video is a challenging problem that requires effective means to extract both visual and tempo…
UCL（伦敦大学学院）；Università di Verona（维罗纳大学）；Politecnico di Milano（米兰理工大学）
Kinematic trajectories recorded from surgical robots contain information about surgical gestures and potentially encode cues about surgeon's skill levels. Automatic segmentation of these trajectories into meaningful action units could help to develop new metrics for surgical skill assessment as well as to simplify surgical automation. State-of-the-art methods for action recognition relied on manual labelling of large datasets, which is time consuming and error prone. Unsupervised methods have been developed to overcome these limitations. However, they often rely on tedious parameter tuning and…
Argonne National Laboratory（阿贡国家实验室）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Robot-assisted minimally invasive surgery is improving surgeon performance and patient outcomes. This innovation is also turning what has been a subjective practice into motion sequences that can be precisely measured. A growing number of studies have used machine learning to analyze video and kinematic data captured from surgical robots. In these studies, models are typically trained on benchmark datasets for representative surgical tasks to assess surgeon skill levels. While they have shown that novices and experts can be accurately classified, it is not clear whether machine learning can se…
Expert Consensus on Criteria for the Automated Assessment of Laparoscopic Camera Navigation
University Medical Center Göttingen（哥廷根大学医学中心）；Fraunhofer Research Institution for Individualized and Cell-based Medical Engineering (IMTE)（弗劳恩霍夫个性化与细胞医学工程研究所）
The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Imperial College London（帝国理工学院）；National University of Singapore (Suzhou) Research Institute（新加坡国立大学（苏州）研究院）
Surgical captioning plays an important role in surgical instruction prediction and report generation. However, the majority of captioning models still rely on the heavy computational object detector or feature extractor to extract regional features. In addition, the detection model requires additional bounding box annotation which is costly and needs skilled annotators. These lead to inference delay and limit the captioning model to deploy in real-time robotic surgery. For this purpose, we design an end-to-end detector and feature extractor-free captioning model by utilizing the patch-based sh…
Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis
Applied Robotics and AI Solutions (ARAS)（应用机器人与人工智能解决方案）；Faculties of Electrical and Computer Engineering（电气与计算机工程学院）；K.N. Toosi University of Technology（卡里姆·纳尼·托西技术大学）；Translational Ophthalmology Research Center（转化眼科研究中心）；Farabi Eye Hospital（法拉比眼科医院）；Tehran University of Medical Sciences（德黑兰医科大学）；Noor Ophthalmology Research Center（努尔眼科研究中心）；University of Alberta（阿尔伯塔大学）；Departments of Electrical and Computer Engineering & Biomedical Engineering（电气与计算机工程及生物医学工程系）
OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining
Shanghai AI Laboratory（上海人工智能实验室）；University of Strasbourg（斯特拉斯堡大学）；Monash University（莫纳什大学）；University of Michigan（密歇根大学）；Shanghai Jiao Tong University（上海交通大学）；Xiamen University（厦门大学）；The Hong Kong Polytechnic University（香港理工大学）；Technische Universität München（慕尼黑工业大学）
Bournemouth University（伯恩茅斯大学）；Communication University of China（中国传媒大学）
Automatic surgical gesture recognition is a prerequisite of intra-operative computer assistance and objective surgical skill assessment. Prior works either require additional sensors to collect kinematics data or have limitations on capturing temporal information from long and untrimmed surgical videos. To tackle these challenges, we propose a novel temporal convolutional architecture to automatically detect and segment surgical gestures with corresponding boundaries only using RGB videos. We devise our method with a symmetric dilation structure bridged by a self-attention module to encode and…
Uncertainty-aware Self-supervised Learning for Cross-domain Technical Skill Assessment in Robot-assisted Surgery
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Scuola Superiore Sant’Anna（圣安娜高等研究院）；Politecnico di Milano（米兰理工大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Mitigating Biases in Surgical Operating Rooms with Geometry
Computer Aided Medical Procedures, TU Munich, Germany（图宾根大学医学辅助技术部）；Munich Center for Machine Learning, Germany（慕尼黑机器学习中心）；UCL Hawkes Institute, Dept. Computer Science, University College London（伦敦大学学院霍克斯研究所）；Minimally Invasive Surgery, University Hospital of Munich (LMU), Germany（慕尼黑大学医院微创外科）
CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Qilu Hospital of Shandong University（山东大学齐鲁医院）；National University o（国立大学）
Virtual-reality based patient-specific simulation of spine surgical procedures: A fast, highly automated and high-fidelity system for surgical education and planning
Holland Bone and Joint Program, Sunnybrook Research Institute（霍尔德骨科与关节程序，圣布鲁诺研究学院）；Division of Spine Surgery, Sunnybrook Health Sciences Centre（脊柱外科部，圣布鲁诺健康科学中心）；Division of Orthopaedic Surgery, Department of Surgery, University of Toronto（骨科部，外科部，多伦多大学）
