手术活动的实时识别与预测，是提升机器人辅助手术安全性与自主性的基础。本文提出一种多模态 transformer 架构，基于运动学和视频数据的短片段，实时识别并预测手术手势与轨迹。我们开展消融研究，评估融合不同输入模态及其表示对手势识别和预测性能的影响。我们使用 JHU-ISI Gesture and Skill Assessment Working Set（JIGSAWS）数据集对所提架构进行端到端评估。通过有效融合运动学特征与空间及上下文视频特征，我们的模型以 89.5% 的手势预测准确率优于现有最优方法（SOTA）。凭借计算高效的模型，其处理 1 秒输入窗口可达到 1.1–1.3 ms 的实时性能。
英文摘要：
Real-time recognition and prediction of surgical activities are fundamental to advancing safety and autonomy in robot-assisted surgery. This paper presents a multimodal transformer architecture for real-time recognition and prediction of surgical gestures and trajectories based on short segments of kinematic and video data. We conduct an ablation study to evaluate the impact of fusing different input modalities and their representations on gesture recognition and prediction performance. We perform an end-to-end assessment of the proposed architecture using the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) dataset. Our model outperforms the state-of-the-art (SOTA) with 89.5\% accuracy for gesture prediction through effective fusion of kinematic features with spatial and contextual video features. It achieves the real-time performance of 1.1-1.3ms for processing a 1-second input window by relying on a computationally efficient model.
Leveraging Surgical Activity Grammar for Primary Intention Prediction in Laparoscopy Procedures
Huazhong University of Science and Technology（华中科技大学）；Union Hospital, Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院附属协和医院）
SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy
Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Department of Thoracic Surgery, Peking University People’s Hospital（北京大学人民医院胸外科）；Thoracic Oncology Institute, Peking University People’s Hospital（北京大学人民医院胸部肿瘤研究所）；Research Unit of Intelligence Diagnosis and Treatment in Early Non-small Cell Lung Cancer, Chinese Academy of Medical Sciences（中国医学科学院早期非小细胞肺癌智能诊断与治疗研究组）；Institute of Advanced Clinical Medicine, Peking University（北京大学先进临床医学院）；Beijing Key Laboratory of Innovative Application of Big Data in Lung Cancer, Peking University People’s Hospital（北京大学人民医院肺癌大数据创新应用北京市重点实验室）
Datasets are an essential component for training effective machine learning models. In particular, surgical robotic datasets have been key to many advances in semi-autonomous surgeries, skill assessment, and training. Simulated surgical environments can enhance the data collection process by making it faster, simpler and cheaper than real systems. In addition, combining data from multiple robotic domains can provide rich and diverse training data for transfer learning algorithms. In this paper, we present the DESK (Dexterous Surgical Skill) dataset. It comprises a set of surgical robotic skill…
Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglement
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Chair for Computer Aided Medical Procedures, Technical University of Munich（慕尼黑技术大学计算机辅助医疗程序主席职位）；Department of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程系）；Center for Computational and Molecular Biology, Brown University（布朗大学计算与分子生物学中心）
Self-supervised, multi-modal learning has been successful in holistic representation of complex scenarios. This can be useful to consolidate information from multiple modalities which have multiple, versatile uses. Its application in surgical robotics can lead to simultaneously developing a generalised machine understanding of the surgical process and reduce the dependency on quality, expert annotations which are generally difficult to obtain. We develop a self-supervised, multi-modal representation learning paradigm that learns representations for surgical gestures from video and kinematics. …
OSSAR: Towards Open-Set Surgical Activity Recognition in Robot-assisted Surgery
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Beijing Institute of Technology（北京理工大学）；Qilu Hospital of Shandong University（山东大学齐鲁医院）；Tongji University（同济大学）；University College London（伦敦大学学院）
SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment
UCL Hawkes Institute and the Department of Medical Physics & Biomedical Engineering, UCL（UCL哈维斯研究所及UCL医学物理与生物医学工程系）；UCL Hawkes Institute and the Department of Computer Science, UCL（UCL哈维斯研究所及UCL计算机科学系）；UCL Hawkes Institute and the Division of Informatics, Imaging & Data Sciences, The University of Manchester（UCL哈维斯研究所及信息学、成像与数据科学系，曼彻斯特大学）；UCL Hospitals NHS Foundation Trust（UCL医院 NHS基金会信托）；Griffin Institute, Northwick Park and St Mark’s Hospital（格里芬研究所，北wick公园及圣马可医院）
Force estimation using neural networks is a promising approach to enable haptic feedback in minimally invasive surgical robots without end-effector force sensors. Various network architectures have been proposed, but none have been tested in real time with surgical-like manipulations. Thus, questions remain about the real-time transparency and stability of force feedback from neural network-based force estimates. We characterize the real-time impedance transparency and stability of force feedback rendered on a da Vinci Research Kit teleoperated surgical robot using neural networks with vision-…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during R…
Model selection and real-time skill assessment for suturing in robotic surgery
Department of Mechanical Engineering, Imperial College London（帝国理工学院机械工程系）；Department of Computing, Imperial College London（帝国理工学院计算机系）；Department of Mechanical Engineering, Stanford University（斯坦福大学机械工程系）；Intuitive Surgical, Inc.（Intuitive Surgical公司）；Department of Bioengineering, Imperial College London（帝国理工学院生物工程系）
Surgical Re-enactment for Operating Room Workflow Datasets
TUM University Hospital（慕尼黑工业大学附属医院）；TUM School of Medicine and Health（慕尼黑工业大学医学与健康学院）；TUM School of Computation, Information, and Technology（慕尼黑工业大学计算、信息与技术学院）；University of Alberta（阿尔伯塔大学）
Purpose: This paper focuses on an automated analysis of surgical motion profiles for objective skill assessment and task recognition in robot-assisted surgery. Existing techniques heavily rely on conventional statistic measures or shallow modelings based on hand-engineered features and gesture segmentation. Such developments require significant expert knowledge, are prone to errors, and are less efficient in online adaptive training systems. Methods: In this work, we present an efficient analytic framework with a parallel deep learning architecture, SATR-DL, to assess trainee expertise and rec…
To fluently collaborate with people, robots need the ability to recognize human activities accurately. Although modern robots are equipped with various sensors, robust human activity recognition (HAR) still remains a challenging task for robots due to difficulties related to multimodal data fusion. To address these challenges, in this work, we introduce a deep neural network-based multimodal HAR algorithm, HAMLET. HAMLET incorporates a hierarchical architecture, where the lower layer encodes spatio-temporal features from unimodal data by adopting a multi-head self-attention mechanism. We devel…
Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery
National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong (CUHK)（香港中文大学）；University College London（伦敦大学学院）；Shun Hing Institute of Advanced Engineering（信兴高等工程研究所）
We present a simple, yet effective and flexible method for action recognition supporting multiple sensor modalities. Multivariate signal sequences are encoded in an image and are then classified using a recently proposed EfficientNet CNN architecture. Our focus was to find an approach that generalizes well across different sensor modalities without specific adaptions while still achieving good results. We apply our method to 4 action recognition datasets containing skeleton sequences, inertial and motion capturing measurements as well as \wifi fingerprints that range up to 120 action classes. …
