Modeling and recognition of surgical activities poses an interesting research problem. Although a number of recent works studied automatic recognition of surgical activities, generalizability of these works across different tasks and different datasets remains a challenge. We introduce a modality that is robust to scene variation, and that is able to infer part information such as orientational and relative spatial relationships. The proposed modality is based on spatial temporal graph representations of surgical tools in videos, for surgical activity recognition. To explore its effectiveness, we model and recognize surgical gestures with the proposed modality. We construct spatial graphs connecting the joint pose estimations of surgical tools. Then, we connect each joint to the corresponding joint in the consecutive frames forming inter-frame edges representing the trajectory of the joint over time. We then learn hierarchical spatial temporal graph representations using Spatial Temporal Graph Convolutional Networks (ST-GCN). Our experiments show that learned spatial temporal graph representations perform well in surgical gesture recognition even when used individually. We experiment with the Suturing task of the JIGSAWS dataset where the chance baseline for gesture recognition is 10%. Our results demonstrate 68% average accuracy which suggests a significant improvement. Learned hierarchical spatial temporal graph representations can be used either individually, in cascades or as a complementary modality in surgical activity recognition, therefore provide a benchmark for future studies. To our knowledge, our paper is the first to use spatial temporal graph representations of surgical tools, and pose-based skeleton representations in general, for surgical activity recognition.
Automatic surgical activity recognition enables more intelligent surgical devices and a more efficient workflow. Integration of such technology in new operating rooms has the potential to improve care delivery to patients and decrease costs. Recent works have achieved a promising performance on surgical activity recognition; however, the lack of generalizability of these models is one of the critical barriers to the wide-scale adoption of this technology. In this work, we study the generalizability of surgical activity recognition models across operating rooms. We propose a new domain adaptati…
Activity recognition in surgical videos is a key research area for developing next-generation devices and workflow monitoring systems. Since surgeries are long processes with highly-variable lengths, deep learning models used for surgical videos often consist of a two-stage setup using a backbone and temporal sequence model. In this paper, we investigate many state-of-the-art backbones and temporal models to find architectures that yield the strongest performance for surgical activity recognition. We first benchmark the models performance on a large-scale activity recognition dataset containin…
OSSAR: Towards Open-Set Surgical Activity Recognition in Robot-assisted Surgery
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Beijing Institute of Technology（北京理工大学）；Qilu Hospital of Shandong University（山东大学齐鲁医院）；Tongji University（同济大学）；University College London（伦敦大学学院）
SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning
Intuitive Surgical, Inc.（Intuitive Surgical公司）；Center for Research and Formation in Artificial Intelligence (CinfonIA), Universidad de los Andes（人工智能研究中心（CinfonIA））
Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis
Applied Robotics and AI Solutions (ARAS)（应用机器人与人工智能解决方案）；Faculties of Electrical and Computer Engineering（电气与计算机工程学院）；K.N. Toosi University of Technology（卡里姆·纳尼·托西技术大学）；Translational Ophthalmology Research Center（转化眼科研究中心）；Farabi Eye Hospital（法拉比眼科医院）；Tehran University of Medical Sciences（德黑兰医科大学）；Noor Ophthalmology Research Center（努尔眼科研究中心）；University of Alberta（阿尔伯塔大学）；Departments of Electrical and Computer Engineering & Biomedical Engineering（电气与计算机工程及生物医学工程系）
CholecTriplet2021: A benchmark challenge for surgical action triplet recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Chicago（芝加哥大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Hefei University of Technology（合肥工业大学）；Shanghai Jiao Tong University（上海交通大学）；Argonne National Laboratory（阿贡国家实验室）；Medtronic（美敦力）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
BenchHAR: Benchmarking Self-Supervised Learning for Generalizable Sensor-based Activity Recognition
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Peking University（北京大学）；Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工联合研究技术联盟）
Beyond Role-Based Surgical Domain Modeling: Generalizable Re-Identification in the Operating Room
Chair for Computer Aided Medical Procedures, Technical University of Munich（计算机辅助医疗程序研究所，慕尼黑技术大学）；UCL Hawkes Institute, Dept. Computer Science, University College London（UCL哈维斯研究所，伦敦大学学院计算机科学系）；Department of General, Visceral, and Transplant Surgery, University Hospital, Ludwig-Maximilians-University（普通外科、 visceral外科及移植外科部门，路德维希-马克西米利安大学医院）
ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection
School of Nursing, The Hong Kong Polytechnic University（香港理工大学护理学院）；Nanfang Hospital, Southern Medical University（南方医科大学南芳医院）；Department of Surgery, The Chinese University of Hong Kong（香港中文大学外科部）；South China University of Technology（华南理工大学）；Singapore Management University（新加坡国立管理学院）
Toward Real-Time Surgical Scene Segmentation via a Spike-Driven Video Transformer with Spike-Informed Pretraining
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Alberta（阿尔伯塔大学）；School of Medicine, Yale University（耶鲁大学医学院）；Southern University of Science and Technology（南方科技大学）；Nanfang Hospital Southern Medical University（南方医科大学南华医院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）；Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology（深圳先进技术研究院计算机科学与控制工程学院）
Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgery
University of Basel（巴塞尔大学）；University of Strasbourg（斯特拉斯堡大学）；University of Verona（维罗纳大学）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；Inselspital Bern University Hospital（伯尔尼大学医院）；IRCAD France（法国消化外科和腹腔镜影像研究所）
Surgical Re-enactment for Operating Room Workflow Datasets
TUM University Hospital（慕尼黑工业大学附属医院）；TUM School of Medicine and Health（慕尼黑工业大学医学与健康学院）；TUM School of Computation, Information, and Technology（慕尼黑工业大学计算、信息与技术学院）；University of Alberta（阿尔伯塔大学）
Tecnologico de Monterrey（蒙特雷理工学院）；University of Leeds（利兹大学）
Surgical tool detection in minimally invasive surgery is an essential part of computer-assisted interventions. Current approaches are mostly based on supervised methods which require large fully labeled data to train supervised models and suffer from pseudo label bias because of class imbalance issues. However large image datasets with bounding box annotations are often scarcely available. Semi-supervised learning (SSL) has recently emerged as a means for training large models using only a modest amount of annotated data; apart from reducing the annotation cost. SSL has also shown promise to p…
When do they StOP?: A First Step Towards Automatically Identifying Team Communication in the Operating Room
University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Bern（伯尔尼大学）；Clarunis（克拉鲁尼斯）；University of Basel（巴塞尔大学）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；Bern University Hospital（伯尔尼大学医院）
Georgia Institute of Technology（佐治亚理工学院）；University of Southern California（南加利福尼亚大学）；Intuitive Surgical Inc.（直觉外科手术公司）
Adverse surgical outcomes are costly to patients and hospitals. Approaches to benchmark surgical care are often limited to gross measures across the entire procedure despite the performance of particular tasks being largely responsible for undesirable outcomes. In order to produce metrics from tasks as opposed to the whole procedure, methods to recognize automatically individual surgical tasks are needed. In this paper, we propose several approaches to recognize surgical activities in robot-assisted minimally invasive surgery using deep learning. We collected a clinical dataset of 100 robot-as…
Automatic recognition of surgical activities in the operating room (OR) is a key technology for creating next generation intelligent surgical devices and workflow monitoring/support systems. Such systems can potentially enhance efficiency in the OR, resulting in lower costs and improved care delivery to the patients. In this paper, we investigate automatic surgical activity recognition in robot-assisted operations. We collect the first large-scale dataset including 400 full-length multi-perspective videos from a variety of robotic surgery cases captured using Time-of-Flight cameras. We densely…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
Previous work on surgical skill assessment using intraoperative tool motion in the operating room (OR) has focused on highly-structured surgical tasks such as cholecystectomy. Further, these methods only considered generic motion metrics such as time and number of movements, which are of limited instructive value. In this paper, we developed and evaluated an automated approach to the surgical skill assessment of nasal septoplasty in the OR. The obstructed field of view and highly unstructured nature of septoplasty precludes trainees from efficiently learning the procedure. We propose a descrip…
Prior work has demonstrated the feasibility of automated activity recognition in robot-assisted surgery from motion data. However, these efforts have assumed the availability of a large number of densely-annotated sequences, which must be provided manually by experts. This process is tedious, expensive, and error-prone. In this paper, we present the first analysis under the assumption of scarce annotations, where as little as one annotated sequence is available for training. We demonstrate feasibility of automated recognition in this challenging setting, and we show that learning representatio…
Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Intuitive Surgical Inc.（直觉外科公司）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition
University of Verona（维罗纳大学）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IRCAD（斯特拉斯堡微创外科研究中心）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学医院基金会IRCCS）
Modeling and automatically recognizing surgical activities are fundamental steps toward automation in surgery and play important roles in providing timely feedback to surgeons. Accurately recognizing surgical activities in video poses a challenging problem that requires an effective means of learning both spatial and temporal dynamics. Human gaze and visual saliency carry important information about visual attention and can be used to extract more relevant features that better reflect these spatial and temporal dynamics. In this study, we propose to use human gaze with a spatio-temporal attent…
Federated Learning for Surgical Vision in Appendicitis Classification: Results of the FedSurg EndoVis 2024 Challenge
Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（转化外科肿瘤学部，肿瘤疾病国家中心（NCT），NCT/UCC德累斯顿，DKFZ、医学院和卡尔·戈斯瓦尔德·卡尔医院之间的合作）；Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology（人机协同触觉互联网中心（CeTI），德累斯顿技术大学）；Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（医学部I，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg（肿瘤医学部，肿瘤疾病国家中心（NCT），海德堡大学医院）；Weldon School of Biomedical Engineering, Purdue University（生物医学工程学院，普渡大学）；Else Kroener Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（Else Kroener Fresenius数字健康中心，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；IHU Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡IHU，影像引导手术研究所）；University of Strasbourg, CNRS, INSERM, ICube, UMR7357（斯特拉斯堡大学，CNRS，INSERM，ICube，UMR7357）；Dr. NTR University of Health Sciences（Dr. NTR健康科学大学）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（visceral、胸腔和血管外科部，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；Digital Technologies, Medtronic（数字技术，美敦力）；UCL Hawkes Institute and Department of Computer Science, University College London（UCL Hawkes研究所和计算机科学系，伦敦大学学院）；German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems（德国癌症研究中心（DKFZ）海德堡，智能医学系统部）；German Cancer Research Center (DKFZ) Heidelberg, Helmholtz Imaging（德国癌症研究中心（DKFZ）海德堡，海德堡成像）；National Center for Tumor Diseases (NCT) Heidelberg, a partnership between DKFZ and Heidelberg University Hospital（肿瘤疾病国家中心（NCT）海德堡，DKFZ和海德堡大学医院之间的合作）；Faculty of Mathematics and Computer Science, Heidelberg University（数学和计算机科学学院，海德堡大学）；Helmholtz Information and Data Science School for Health（海德堡信息与数据科学健康学院）；Heidelberg University Hospital, Surgical Clinic, Surgical AI Research Group（海德堡大学医院，外科诊所，外科AI研究组）；Krankenhaus St. Joseph-Stift Dresden GmbH（德累斯顿圣约瑟夫修道院医院）
College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）；Medical Research, Intuitive Surgical, Inc.（直觉外科公司医学研究部）
Purpose: Surgical task-based metrics (rather than entire procedure metrics) can be used to improve surgeon training and, ultimately, patient care through focused training interventions. Machine learning models to automatically recognize individual tasks or activities are needed to overcome the otherwise manual effort of video review. Traditionally, these models have been evaluated using frame-level accuracy. Here, we propose evaluating surgical activity recognition models by their effect on task-based efficiency metrics. In this way, we can determine when models have achieved adequate performa…
University Hospital of Strasbourg（斯特拉斯堡大学医院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学综合医院基金会IRCCS）；IRCAD（斯特拉斯堡消化癌症研究所）；University of Verona（维罗纳大学）；ICube（斯特拉斯堡大学工程、计算机科学、成像与数学实验室）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）
Purpose: Automatic segmentation and classification of surgical activity is crucial for providing advanced support in computer-assisted interventions and autonomous functionalities in robot-assisted surgeries. Prior works have focused on recognizing either coarse activities, such as phases, or fine-grained activities, such as gestures. This work aims at jointly recognizing two complementary levels of granularity directly from videos, namely phases and steps. Method: We introduce two correlated surgical activities, phases and steps, for the laparoscopic gastric bypass procedure. We propose a Mul…
University of Chinese Academy of Sciences（中国科学院大学）；Microsoft Research Asia（微软亚洲研究院）；Pengcheng Laboratory（鹏城实验室）
Human activity recognition requires the efforts to build a generalizable model using the training datasets with the hope to achieve good performance in test datasets. However, in real applications, the training and testing datasets may have totally different distributions due to various reasons such as different body shapes, acting styles, and habits, damaging the model's generalization performance. While such a distribution gap can be reduced by existing domain adaptation approaches, they typically assume that the test data can be accessed in the training stage, which is not realistic. In thi…
We apply recurrent neural networks to the task of recognizing surgical activities from robot kinematics. Prior work in this area focuses on recognizing short, low-level activities, or gestures, and has been based on variants of hidden Markov models and conditional random fields. In contrast, we work on recognizing both gestures and longer, higher-level activites, or maneuvers, and we model the mapping from kinematics to gestures/maneuvers with recurrent neural networks. To our knowledge, we are the first to apply recurrent neural networks to this task. Using a single model and a single set of …
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube研究中心）；UMR7357（法国大学-研究中心7357）；IHU Strasbourg（斯特拉斯堡IHU医院）；University Digestive Health Care Center – Clarunis（大学消化健康中心 – Clarunis）；Ludwig Maximilian University of Munich（慕尼黑路易斯·马克西米利安大学）；Chinese University of Hong Kong（香港中文大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Improving Out-of-distribution Human Activity Recognition via IMU-Video Cross-modal Representation Learning
Department of Computer Science & Engineering University of Minnesota（计算机科学与工程系明尼苏达大学）；Department of Mechanical Engineering University of Minnesota（机械工程系明尼苏达大学）；Department of Neurosurgery University of Minnesota（神经外科系明尼苏达大学）
ICube（ICube实验室）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IRCAD（法国消化外科研究所）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学综合医院基金会IRCCS）
Recognition of surgical activity is an essential component to develop context-aware decision support for the operating room. In this work, we tackle the recognition of fine-grained activities, modeled as action triplets <instrument, verb, target> representing the tool activity. To this end, we introduce a new laparoscopic dataset, CholecT40, consisting of 40 videos from the public dataset Cholec80 in which all frames have been annotated using 128 triplet classes. Furthermore, we present an approach to recognize these triplets directly from the video data. It relies on a module called Class Act…
EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；School of Advanced Manufacturing, Fuzhou University（福州大学先进制造学院）
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）
Recent advancements in deep learning methods bring computer-assistance a step closer to fulfilling promises of safer surgical procedures. However, the generalizability of such methods is often dependent on training on diverse datasets from multiple medical institutions, which is a restrictive requirement considering the sensitive nature of medical data. Recently proposed collaborative learning methods such as Federated Learning (FL) allow for training on remote datasets without the need to explicitly share data. Even so, data annotation still represents a bottleneck, particularly in medicine a…
Long-term complex activity recognition and localisation can be crucial for decision making in autonomous systems such as smart cars and surgical robots. Here we address the problem via a novel deformable, spatiotemporal scene graph approach, consisting of three main building blocks: (i) action tube detection, (ii) the modelling of the deformable geometry of parts, and (iii) a graph convolutional network. Firstly, action tubes are detected in a series of snippets. Next, a new 3D deformable RoI pooling layer is designed for learning the flexible, deformable geometry of the constituent action tub…
SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Pavia（帕维亚大学）；University Digestive Health Care Center – Clarunis（克拉鲁尼斯大学消化保健中心）；Università di Roma Tor Vergata（罗马第二大学）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；University Hospital Zurich（苏黎世大学医院）；University Hospital Bern（伯尔尼大学医院）；University of Basel（巴塞尔大学）
Purpose: This paper focuses on an automated analysis of surgical motion profiles for objective skill assessment and task recognition in robot-assisted surgery. Existing techniques heavily rely on conventional statistic measures or shallow modelings based on hand-engineered features and gesture segmentation. Such developments require significant expert knowledge, are prone to errors, and are less efficient in online adaptive training systems. Methods: In this work, we present an efficient analytic framework with a parallel deep learning architecture, SATR-DL, to assess trainee expertise and rec…
Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes
ROCS（罗克辛研究所）；Balgrist University Hospital（巴尔格里斯大学医院）；University of Zurich（苏黎世大学）；Dept. of Orthopedics（骨科部门）；Computer Vision and Geometry Group（计算机视觉与几何组）；ETH Zurich（苏黎世联邦理工学院）
Towards Completeness: A Generalizable Action Proposal Generator for Zero-Shot Temporal Action Localization
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（教育部机器智能与先进计算重点实验室）；Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University（广东省信息安全技术重点实验室（中山大学））
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）
Objective: Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classif…
There are many realistic applications of activity recognition where the set of potential activity descriptions is combinatorially large. This makes end-to-end supervised training of a recognition system impractical as no training set is practically able to encompass the entire label set. In this paper, we present an approach to fine-grained recognition that models activities as compositions of dynamic action signatures. This compositional approach allows us to reframe fine-grained recognition as zero-shot activity recognition, where a detector is composed "on the fly" from simple first-princip…
IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Georgia Institute of Technology（佐治亚理工学院）；College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）；School of Interactive Computing, Georgia Institute of Technology（佐治亚理工学院交互计算学院）；Emory University（埃默里大学）
