Hand hygiene is one of the most significant factors in preventing hospital acquired infections (HAI) which often be transmitted by medical staffs in contact with patients in the operating room (OR). Hand hygiene monitoring could be important to investigate and reduce the outbreak of infections within the OR. However, an effective monitoring tool for hand hygiene compliance is difficult to develop due to the visual complexity of the OR scene. Recent progress in video understanding with convolutional neural net (CNN) has increased the application of recognition and detection of human actions. Leveraging this progress, we proposed a fully automated hand hygiene monitoring tool of the alcohol-based hand rubbing action of anesthesiologists on OR video using spatio-temporal features with 3D CNN. First, the region of interest (ROI) of anesthesiologists' upper body were detected and cropped. A temporal smoothing filter was applied to the ROIs. Then, the ROIs were given to a 3D CNN and classified into two classes: rubbing hands or other actions. We observed that a transfer learning from Kinetics-400 is beneficial and the optical flow stream was not helpful in our dataset. The final accuracy, precision, recall and F1 score in testing is 0.76, 0.85, 0.65 and 0.74, respectively.
Automatic recognition of surgical activities in the operating room (OR) is a key technology for creating next generation intelligent surgical devices and workflow monitoring/support systems. Such systems can potentially enhance efficiency in the OR, resulting in lower costs and improved care delivery to the patients. In this paper, we investigate automatic surgical activity recognition in robot-assisted operations. We collect the first large-scale dataset including 400 full-length multi-perspective videos from a variety of robotic surgery cases captured using Time-of-Flight cameras. We densely…
Sterilizable Scene Graph Generation for Operating Rooms
Technical University of Darmstadt（达姆施塔特工业大学）；ImFusion GmbH（ImFusion有限公司）；Carl Zeiss AG（卡尔蔡司股份公司）；University Medical Center Mainz（美因茨大学医学中心）；Inria Center at University Côte d’Azur（蔚蓝海岸大学法国国家信息与自动化研究所中心）
Automatic surgical activity recognition enables more intelligent surgical devices and a more efficient workflow. Integration of such technology in new operating rooms has the potential to improve care delivery to patients and decrease costs. Recent works have achieved a promising performance on surgical activity recognition; however, the lack of generalizability of these models is one of the critical barriers to the wide-scale adoption of this technology. In this work, we study the generalizability of surgical activity recognition models across operating rooms. We propose a new domain adaptati…
ICube, University of Strasbourg, CNRS, IHU Strasbourg（ICube实验室，斯特拉斯堡大学，法国国家科学研究中心，斯特拉斯堡大学医院）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Many approaches have been proposed for human pose estimation in single and multi-view RGB images. However, some environments, such as the operating room, are still very challenging for state-of-the-art RGB methods. In this paper, we propose an approach for multi-view 3D human pose estimation from RGB-D images and demonstrate the benefits of using the additional depth channel for pose refinement beyond its use for the generation of improved features. The proposed method permits the joint detection and estimation of the poses without knowing a priori the number of persons present in the scene. W…
Data-driven approaches to assist operating room (OR) workflow analysis depend on large curated datasets that are time consuming and expensive to collect. On the other hand, we see a recent paradigm shift from supervised learning to self-supervised and/or unsupervised learning approaches that can learn representations from unlabeled datasets. In this paper, we leverage the unlabeled data captured in robotic surgery ORs and propose a novel way to fuse the multi-modal data for a single video frame or image. Instead of producing different augmentations (or 'views') of the same image or video frame…
TrackOR: Towards Personalized Intelligent Operating Rooms Through Robust Tracking
Computer Aided Medical Procedures, TU Munich, Germany（图宾根大学医学辅助技术中心）；Munich Center for Machine Learning, Germany（慕尼黑机器学习中心）；Minimally Invasive Surgery, University Hospital of Munich (LMU), Germany（慕尼黑大学微创手术中心）
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
Previous work on surgical skill assessment using intraoperative tool motion in the operating room (OR) has focused on highly-structured surgical tasks such as cholecystectomy. Further, these methods only considered generic motion metrics such as time and number of movements, which are of limited instructive value. In this paper, we developed and evaluated an automated approach to the surgical skill assessment of nasal septoplasty in the OR. The obstructed field of view and highly unstructured nature of septoplasty precludes trainees from efficiently learning the procedure. We propose a descrip…
ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube实验室，法国国家科学研究中心，斯特拉斯堡大学医院）；Radiology Department, University Hospital of Strasbourg（斯特拉斯堡大学医院放射科）
Person detection and pose estimation is a key requirement to develop intelligent context-aware assistance systems. To foster the development of human pose estimation methods and their applications in the Operating Room (OR), we release the Multi-View Operating Room (MVOR) dataset, the first public dataset recorded during real clinical interventions. It consists of 732 synchronized multi-view frames recorded by three RGB-D cameras in a hybrid OR. It also includes the visual challenges present in such environments, such as occlusions and clutter. We provide camera calibration parameters, color a…
Isfahan University of Technology（伊斯法罕理工大学）；University of Michigan（密歇根大学）
Hand gesture is one of the most important means of touchless communication between human and machines. There is a great interest for commanding electronic equipment in surgery rooms by hand gesture for reducing the time of surgery and the potential for infection. There are challenges in implementation of a hand gesture recognition system. It has to fulfill requirements such as high accuracy and fast response. In this paper we introduce a system of hand gesture recognition based on a deep learning approach. Deep learning is known as an accurate detection model, but its high complexity prevents …
Oxford Brookes University（牛津布鲁克斯大学）；Imperial College London（帝国理工学院）
Nowadays, there are more surgical procedures that are being performed using minimally invasive surgery (MIS). This is due to its many benefits, such as minimal post-operative problems, less bleeding, minor scarring, and a speedy recovery. However, the MIS's constrained field of view, small operating room, and indirect viewing of the operating scene could lead to surgical tools colliding and potentially harming human organs or tissues. Therefore, MIS problems can be considerably reduced, and surgical procedure accuracy and success rates can be increased by using an endoscopic video feed to dete…
Modern surgeries are performed in complex and dynamic settings, including ever-changing interactions between medical staff, patients, and equipment. The holistic modeling of the operating room (OR) is, therefore, a challenging but essential task, with the potential to optimize the performance of surgical teams and aid in developing new surgical technologies to improve patient outcomes. The holistic representation of surgical scenes as semantic scene graphs (SGG), where entities are represented as nodes and relations between them as edges, is a promising direction for fine-grained semantic OR u…
University Hospital of Strasbourg（斯特拉斯堡大学医院）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）
Purpose: Face detection is a needed component for the automatic analysis and assistance of human activities during surgical procedures. Efficient face detection algorithms can indeed help to detect and identify the persons present in the room, and also be used to automatically anonymize the data. However, current algorithms trained on natural images do not generalize well to the operating room (OR) images. In this work, we provide a comparison of state-of-the-art face detectors on OR data and also present an approach to train a face detector for the OR by exploiting non-annotated OR images. Me…
Johns Hopkins University（约翰斯·霍普金斯大学）；Georgia Institute of Technology（佐治亚理工学院）；Université de Montréal（蒙特利尔大学）；Intuitive Surgical Inc.（直观外科公司）
Purpose: We describe a 3D multi-view perception system for the da Vinci surgical system to enable Operating room (OR) scene understanding and context awareness. Methods: Our proposed system is comprised of four Time-of-Flight (ToF) cameras rigidly attached to strategic locations on the daVinci Xi patient side cart (PSC). The cameras are registered to the robot's kinematic chain by performing a one-time calibration routine and therefore, information from all cameras can be fused and represented in one common coordinate frame. Based on this architecture, a multi-view 3D scene semantic segmentati…
Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Intuitive Surgical Inc.（直觉外科公司）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms
The Chinese University of Hong Kong（香港中文大学）；Technical University of Denmark（丹麦技术大学）；Huazhong University of Science and Technology（华中科技大学）；National University of Singapore（新加坡国立大学）
Mitigating Biases in Surgical Operating Rooms with Geometry
Computer Aided Medical Procedures, TU Munich, Germany（图宾根大学医学辅助技术部）；Munich Center for Machine Learning, Germany（慕尼黑机器学习中心）；UCL Hawkes Institute, Dept. Computer Science, University College London（伦敦大学学院霍克斯研究所）；Minimally Invasive Surgery, University Hospital of Munich (LMU), Germany（慕尼黑大学医院微创外科）
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；University of Heidelberg（海德堡大学）；National Center for Tumor Diseases, Partner Site Dresden（国家肿瘤疾病中心德累斯顿分中心）；HIDSS4Health – Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；University of Calabria（卡拉布里亚大学）；understandAI GmbH（understandAI 有限公司）；University of Tübingen（蒂宾根大学）；BMW Group（宝马集团）；KARL STORZ SE & Co. KG（卡尔史托斯公司）；TU Dresden（德累斯顿工业大学）
Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a speci…
Video capture in the surgical operating room (OR) is increasingly possible and has potential for use with computer assisted interventions (CAI), surgical data science and within smart OR integration. Captured video innately carries sensitive information that should not be completely visible in order to preserve the patient's and the clinical teams' identities. When surgical video streams are stored on a server, the videos must be anonymized prior to storage if taken outside of the hospital. In this article, we describe how a deep learning model, Faster R-CNN, can be used for this purpose and h…
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）
The fine-grained localization of clinicians in the operating room (OR) is a key component to design the new generation of OR support systems. Computer vision models for person pixel-based segmentation and body-keypoints detection are needed to better understand the clinical activities and the spatial layout of the OR. This is challenging, not only because OR images are very different from traditional vision datasets, but also because data and annotations are hard to collect and generate in the OR due to privacy concerns. To address these concerns, we first study how joint person pose estimatio…
Surgical Re-enactment for Operating Room Workflow Datasets
TUM University Hospital（慕尼黑工业大学附属医院）；TUM School of Medicine and Health（慕尼黑工业大学医学与健康学院）；TUM School of Computation, Information, and Technology（慕尼黑工业大学计算、信息与技术学院）；University of Alberta（阿尔伯塔大学）
Beyond Role-Based Surgical Domain Modeling: Generalizable Re-Identification in the Operating Room
Chair for Computer Aided Medical Procedures, Technical University of Munich（计算机辅助医疗程序研究所，慕尼黑技术大学）；UCL Hawkes Institute, Dept. Computer Science, University College London（UCL哈维斯研究所，伦敦大学学院计算机科学系）；Department of General, Visceral, and Transplant Surgery, University Hospital, Ludwig-Maximilians-University（普通外科、 visceral外科及移植外科部门，路德维希-马克西米利安大学医院）
Self-Supervised Uncalibrated Multi-View Video Anonymization in the Operating Room
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家卫生研究院，ICube，UMR7357，法国）；University Hospital of Strasbourg, 67000 Strasbourg, France（斯特拉斯堡大学医院，法国）
TeleOR: Real-time Telemedicine System for Full-Scene Operating Room
School of Public Health, Zhejiang University（浙江大学公共卫生学院）；The Second Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第二医院）；Institute of Wenzhou, Zhejiang University（浙江大学温州研究院）；School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Notre Dame（圣母大学）
Geometry OR Tracker: Universal Geometric Operating Room Tracking
Centre for Artificial Intelligence and Robotics（人工智能与机器人中心）；Hong Kong Institute of Science and Innovation Chinese Academy of Sciences（香港科学与创新研究院中国科学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；Institute of Automation Chinese Academy of Sciences (CASIA)（自动化研究所中国科学院（CASIA））；Peking University（北京大学）；University of Trento, Trento, Italy（意大利特伦特大学）；Alibaba DAMO Academy（阿里达摩院）；Xiaomi Inc.（小米公司）；Technische Universität München（慕尼黑技术大学）
Geometry OR Tracker通过多视角几何校正和鲁棒3D点跟踪提升手术室多视角3D跟踪的几何一致性与准确性。
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、法国）；Department for Biomedical Research (DBMR), University of Bern, 3008 Bern, Switzerland（生物医学研究部（DBMR）、伯尔尼大学、瑞士伯尔尼3008、瑞士）；University Hospital of Strasbourg, 67000 Strasbourg, France（斯特拉斯堡大学医院、法国斯特拉斯堡67000、法国）；Department for Visceral Surgery（visceral surgery 部）；Medicine, Bern University Hospital, University of Bern, 3010 Bern, Switzerland（医学、伯尔尼大学医院、伯尔尼大学、瑞士伯尔尼3010、瑞士）
When do they StOP?: A First Step Towards Automatically Identifying Team Communication in the Operating Room
University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Bern（伯尔尼大学）；Clarunis（克拉鲁尼斯）；University of Basel（巴塞尔大学）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；Bern University Hospital（伯尔尼大学医院）
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学ICube实验室，法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Reliable human pose estimation (HPE) is essential to many clinical applications, such as surgical workflow analysis, radiation safety monitoring and human-robot cooperation. Proposed methods for the operating room (OR) rely either on foreground estimation using a multi-camera system, which is a challenge in real ORs due to color similarities and frequent illumination changes, or on wearable sensors or markers, which are invasive and therefore difficult to introduce in the room. Instead, we propose a novel approach based on Pictorial Structures (PS) and on RGB-D data, which can be easily deploy…
OR-VSKC: Resolving Visual-Semantic Knowledge Conflicts in Operating Rooms with Synthetic Data-Guided Alignment
Shanghai University of Engineering Science（上海工程技术大学）；Tencent YouTu Lab（腾讯优图实验室）；Clinical Research Unit, Zhongshan Hospital of Fudan University（复旦大学中山医院临床研究部）
Real-Time Brain Tumor Detection in Intraoperative Ultrasound Using YOLO11: From Model Training to Deployment in the Operating Room
Department of Neurosurgery, Rio Hortega University Hospital（里奥霍尔特加大学医院神经外科部）；Department of Neurosurgery, Tata Memorial Centre, Homi Bhabha National Institute（塔塔纪念中心神经外科部，霍米·巴布哈国家研究所）；Biomedical Engineering Group, University of Valladolid（瓦拉多利德大学生物医学工程小组）；UMR 1253, iBrain, Université de Tours, Inserm（UMR 1253，iBrain， Tours大学，Inserm）；Department of Neurosurgery, CHRU de Tours（图尔CHRU医院神经外科部）；Department of Neurosrugery, ARNAS Civico Di Cristina Benfratelli Hospital（ARNAS公民迪克里斯塔弗拉蒂医院神经外科部）；Fondazione IRCCS Istituto Neurologico Carlo Besta（IRCCS神经学研究所卡尔罗·贝斯塔基金会）；University of Milan（米兰大学）；Johns Hopkins Medical School（约翰霍普金斯医学院）
SurgBox: Agent-Driven Operating Room Sandbox with Surgery Copilot
CAIR, HKISI, CAS（中国科学院香港中文大学信息安全研究所智能计算研究中心）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；Peking Union Medical College Hospital（北京协和医院）
