Real-time surgical phase recognition is a fundamental task in modern operating rooms. Previous works tackle this task relying on architectures arranged in spatio-temporal order, however, the supportive benefits of intermediate spatial features are not considered. In this paper, we introduce, for the first time in surgical workflow analysis, Transformer to reconsider the ignored complementary effects of spatial and temporal features for accurate surgical phase recognition. Our hybrid embedding aggregation Transformer fuses cleverly designed spatial and temporal embeddings by allowing for active queries based on spatial information from temporal embedding sequences. More importantly, our framework processes the hybrid embeddings in parallel to achieve a high inference speed. Our method is thoroughly validated on two large surgical video datasets, i.e., Cholec80 and M2CAI16 Challenge datasets, and outperforms the state-of-the-art approaches at a processing speed of 91 fps.
University College London（伦敦大学学院）；University Hospital Leuven（鲁汶大学医院）
In minimally invasive surgery, surgical workflow segmentation from video analysis is a well studied topic. The conventional approach defines it as a multi-class classification problem, where individual video frames are attributed a surgical phase label. We introduce a novel reinforcement learning formulation for offline phase transition retrieval. Instead of attempting to classify every video frame, we identify the timestamp of each phase transition. By construction, our model does not produce spurious and noisy phase transitions, but contiguous phase blocks. We investigate two different confi…
LoViT: Long Video Transformer for Surgical Phase Recognition
Department of Surgical & Interventional Engineering, King’s College London（外科与介入工程系，伦敦国王学院）；Department of Peter Gorer（彼得·戈尔德系）；Department of Immunobiology, King’s College London（免疫生物学系，伦敦国王学院）
In this paper we introduce OperA, a transformer-based model that accurately predicts surgical phases from long video sequences. A novel attention regularization loss encourages the model to focus on high-quality frames during training. Moreover, the attention weights are utilized to identify characteristic high attention frames for each surgical phase, which could further be used for surgery summarization. OperA is thoroughly evaluated on two datasets of laparoscopic cholecystectomy videos, outperforming various state-of-the-art temporal refinement approaches.
Phase recognition plays an essential role for surgical workflow analysis in computer assisted intervention. Transformer, originally proposed for sequential data modeling in natural language processing, has been successfully applied to surgical phase recognition. Existing works based on transformer mainly focus on modeling attention dependency, without introducing auto-regression. In this work, an Auto-Regressive Surgical Transformer, referred as ARST, is first proposed for on-line surgical phase recognition from laparoscopic videos, modeling the inter-phase correlation implicitly by conditiona…
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipel…
Prior work demonstrated the ability of machine learning to automatically recognize surgical workflow steps from videos. However, these studies focused on only a single type of procedure. In this work, we analyze, for the first time, surgical step recognition on four different laparoscopic surgeries: Cholecystectomy, Right Hemicolectomy, Sleeve Gastrectomy, and Appendectomy. Inspired by the traditional apprenticeship model, in which surgical training is based on the Halstedian method, we paraphrase the "see one, do one, teach one" approach for the surgical intelligence domain as "train one, cla…
Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家卫生研究院，ICube，UMR7357，法国斯特拉斯堡）；University Digestive Health Care Center – Clarunis, 4002 Basel, Switzerland（消化健康研究中心–Clarunis，瑞士巴塞尔）
HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Technische Universität München（慕尼黑工业大学）
Bournemouth University（伯恩茅斯大学）；Technical University of Munich（慕尼黑工业大学）
Automatic surgical instruction generation is a prerequisite towards intra-operative context-aware surgical assistance. However, generating instructions from surgical scenes is challenging, as it requires jointly understanding the surgical activity of current view and modelling relationships between visual information and textual description. Inspired by the neural machine translation and imaging captioning tasks in open domain, we introduce a transformer-backboned encoder-decoder network with self-critical reinforcement learning to generate instructions from surgical images. We evaluate the ef…
Center for Research and Formation in Artificial Intelligence, Universidad de los Andes（安第斯大学人工智能研究与培训中心）；Fundación Santafé de Bogotá（波哥大圣菲基金会）；University of Washington（华盛顿大学）
Automatic recognition of surgical phases in surgical videos is a fundamental task in surgical workflow analysis. In this report, we propose a Transformer-based method that utilizes calibrated confidence scores for a 2-stage inference pipeline, which dynamically switches between a baseline model and a separately trained transition model depending on the calibrated confidence level. Our method outperforms the baseline model on the Cholec80 dataset, and can be applied to a variety of action segmentation methods.
HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Department of General Surgery, Beijing Friendship Hospital, Capital Medical University（首都医科大学附属北京友谊医院普通外科）；Chinese PLA General Hospital（中国人民解放军总医院）
SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognition
Hong Kong University of Science and Technology（香港科技大学）；Tongji University（同济大学）；Tongji Hospital（同济医院）；Tongji Institute of Digestive Disease（同济消化病研究所）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
National University of Singapore（新加坡国立大学）；Imperial College London（帝国理工学院）；National Institute of Technology, Tiruchirappalli（蒂鲁吉拉伯利国家理工学院）；Chinese University of Hong Kong（香港中文大学）；Shun Hing Institute of Advanced Engineering（信兴高等工程研究所）
Visual question answering (VQA) in surgery is largely unexplored. Expert surgeons are scarce and are often overloaded with clinical and academic workloads. This overload often limits their time answering questionnaires from patients, medical students or junior residents related to surgical procedures. At times, students and junior residents also refrain from asking too many questions during classes to reduce disruption. While computer-aided simulators and recording of past surgical procedures have been made available for them to observe and improve their skills, they still hugely rely on medic…
NCT/UCC Dresden（德累斯顿国家肿瘤疾病中心/大学癌症中心）；CeTI, TU Dresden（德累斯顿工业大学触觉互联网卓越中心）；University Hospital Dresden（德累斯顿大学医院）
Research in unpaired video translation has mainly focused on short-term temporal consistency by conditioning on neighboring frames. However for transfer from simulated to photorealistic sequences, available information on the underlying geometry offers potential for achieving global consistency across views. We propose a novel approach which combines unpaired image translation with neural rendering to transfer simulated to photorealistic surgical abdominal scenes. By introducing global learnable textures and a lighting-invariant view-consistency loss, our method produces consistent translation…
MIT（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Olympus Corporation（奥林巴斯公司）
Comprehension of surgical workflow is the foundation upon which artificial intelligence (AI) and machine learning (ML) holds the potential to assist intraoperative decision-making and risk mitigation. In this work, we move beyond mere identification of past surgical phases, into the prediction of future surgical steps and specification of the transitions between them. We use a novel Generative Adversarial Network (GAN) formulation to sample future surgical phases trajectories conditioned on past video frames from laparoscopic cholecystectomy (LC) videos and compare it to state-of-the-art appro…
The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Imperial College London（帝国理工学院）；National University of Singapore (Suzhou) Research Institute（新加坡国立大学（苏州）研究院）
Surgical captioning plays an important role in surgical instruction prediction and report generation. However, the majority of captioning models still rely on the heavy computational object detector or feature extractor to extract regional features. In addition, the detection model requires additional bounding box annotation which is costly and needs skilled annotators. These lead to inference delay and limit the captioning model to deploy in real-time robotic surgery. For this purpose, we design an end-to-end detector and feature extractor-free captioning model by utilizing the patch-based sh…
Purpose: Basic surgical skills of suturing and knot tying are an essential part of medical training. Having an automated system for surgical skills assessment could help save experts time and improve training efficiency. There have been some recent attempts at automated surgical skills assessment using either video analysis or acceleration data. In this paper, we present a novel approach for automated assessment of OSATS based surgical skills and provide an analysis of different features on multi-modal data (video and accelerometer data). Methods: We conduct the largest study, to the best of o…
Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition
The Hong Kong University of Science and Technology（香港科技大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Toward Real-Time Surgical Scene Segmentation via a Spike-Driven Video Transformer with Spike-Informed Pretraining
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Alberta（阿尔伯塔大学）；School of Medicine, Yale University（耶鲁大学医学院）；Southern University of Science and Technology（南方科技大学）；Nanfang Hospital Southern Medical University（南方医科大学南华医院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）；Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology（深圳先进技术研究院计算机科学与控制工程学院）
Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery
National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong (CUHK)（香港中文大学）；University College London（伦敦大学学院）；Shun Hing Institute of Advanced Engineering（信兴高等工程研究所）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Segmentation for tracking surgical instruments plays an important role in robot-assisted surgery. Segmentation of surgical instruments contributes to capturing accurate spatial information for tracking. In this paper, a novel network, Refined Attention Segmentation Network, is proposed to simultaneously segment surgical instruments and identify their categories. The U-shape network which is popular in segmentation is used. Different from previous work, an attention module is adopted to help the network focus on key regions, which can improve the segmentation accuracy. To solve the class imbala…
Pixel-Wise Recognition for Holistic Surgical Scene Understanding
Center for Research and Formation in Artificial Intelligence (CinfonIA)（人工智能研究与形成中心（CinfonIA））；Universidad de los Andes（安第斯大学）；Fundación Santa Fe de Bogotá（圣菲波哥大基金会）
Adaptive transfer learning for surgical tool presence detection in laparoscopic videos through gradual freezing fine-tuning
Institutes of Innovation for Future Society（未来社会创新研究所）；Nagoya University（名古屋大学）；Department of Micro-Nano Mechanical Science and Engineering（微纳米机械科学与工程系）
Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos
Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；University College London（伦敦大学学院）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences（Wellcome/EPSRC介入与外科科学中心）
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space
Zhejiang University（浙江大学）；University of Strasbourg, CNRS, INSERM, ICube, UMR7357（斯特拉斯堡大学，CNRS，INSERM，ICube，UMR7357）；IHU Strasbourg（斯特拉斯堡IHU）；Technical University of Munich（慕尼黑工业大学）
SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning
Intuitive Surgical, Inc.（Intuitive Surgical公司）；Center for Research and Formation in Artificial Intelligence (CinfonIA), Universidad de los Andes（人工智能研究中心（CinfonIA））
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube研究中心）；UMR7357（法国大学-研究中心7357）；IHU Strasbourg（斯特拉斯堡IHU医院）；University Digestive Health Care Center – Clarunis（大学消化健康中心 – Clarunis）；Ludwig Maximilian University of Munich（慕尼黑路易斯·马克西米利安大学）；Chinese University of Hong Kong（香港中文大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary
Computer Aided Medical Procedures (CAMP), TU Munich, Germany Munich Center for Machine Learning (MCML), Munich, Germany University of Strasbourg, France The Chinese University of Hong Kong, Hong Kong
Hierarchical Curriculum Learning for Surgical Video Generation: Focusing on Bilateral Axillo-Breast Approach Robotic Thyroidectomy Datasets
Seoul National University College of Medicine（首尔国立大学医学院）；Seoul National University Hospital（首尔国立大学医院）；YONSEI WITH-YOU Surgery Clinic（延世WITH-YOU外科诊所）；Institute of Convergence Medicine with Innovative Technology, Seoul National University Hospital（首尔国立大学医院创新技术融合医学研究所）
TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation
Birmingham City University（伯明翰城市大学）；University Hospitals Birmingham（伯明翰大学医院）；University of Birmingham（伯明翰大学）；University of the West of England（西英格兰大学）；University of Hertfordshire（赫特福德大学）；University of Bristol（布里斯托大学）；Qatar University（卡塔尔大学）
Activity recognition in surgical videos is a key research area for developing next-generation devices and workflow monitoring systems. Since surgeries are long processes with highly-variable lengths, deep learning models used for surgical videos often consist of a two-stage setup using a backbone and temporal sequence model. In this paper, we investigate many state-of-the-art backbones and temporal models to find architectures that yield the strongest performance for surgical activity recognition. We first benchmark the models performance on a large-scale activity recognition dataset containin…
