Surgical tool segmentation in endoscopic images is the first step towards pose estimation and (sub-)task automation in challenging minimally invasive surgical operations. While many approaches in the literature have shown great results using modern machine learning methods such as convolutional neural networks, the main bottleneck lies in the acquisition of a large number of manually-annotated images for efficient learning. This is especially true in surgical context, where patient-to-patient differences impede the overall generalizability. In order to cope with this lack of annotated data, we propose a self-supervised approach in a robot-assisted context. To our knowledge, the proposed approach is the first to make use of the kinematic model of the robot in order to generate training labels. The core contribution of the paper is to propose an optimization method to obtain good labels for training despite an unknown hand-eye calibration and an imprecise kinematic model. The labels can subsequently be used for fine-tuning a fully-convolutional neural network for pixel-wise classification. As a result, the tool can be segmented in the endoscopic images without needing a single manually-annotated image. Experimental results on phantom and in vivo datasets obtained using a flexible robotized endoscopy system are very promising.
Imperial College London（帝国理工学院）；Shanghai Jiao Tong University（上海交通大学）
Accurate kinematic models are essential for effective control of surgical robots. For tendon driven robots, which is common for minimally invasive surgery, intrinsic nonlinearities are important to consider. Traditional analytical methods allow to build the kinematic model of the system by making certain assumptions and simplifications on the nonlinearities. Machine learning techniques, instead, allow to recover a more complex model based on the acquired data. However, analytical models are more generalisable, but can be over-simplified; data-driven models, on the other hand, can cater for mor…
Chongqing University（重庆大学）；Chongqing University-University of Cincinnati Joint Co-op Institute（重庆大学-辛辛那提大学联合学院）；Imperial College London（帝国理工学院）
Miniaturized instruments are highly needed for robot assisted medical healthcare and treatment, especially for less invasive surgery as it empowers more flexible access to restricted anatomic intervention. But the robotic design is more challenging due to the contradictory needs of miniaturization and the capability of manipulating with a large dexterous workspace. Thus, kinematic parameter optimization is of great significance in this case. To this end, this paper proposes an approach based on dexterous workspace determination for designing a miniaturized tendon-driven surgical instrument und…
Stable Tracking-in-the-Loop Control of Cable-Driven Surgical Manipulators under Erroneous Kinematic Chains
Department of Electrical and Computer Engineering, University of California San Diego（电子工程与计算机科学系，加州大学圣地亚哥分校）；Department of Electrical Engineering and Computer Science, University of Tennessee Knoxville（电子工程与计算机科学系，田纳西大学基洛纳分校）
Overcoming Imperfect Kinematics in Surgical Robotics Through Sim-to-Real Visuomotor Learning
Hamlyn Centre for Robotic Surgery, Institute of Global Health Innovation, Imperial College London（伦敦帝国理工学院全球健康创新研究所哈姆林机器人手术中心）；Department of Surgery and Cancer, Imperial College London（伦敦帝国理工学院外科与癌症系）；Department of Mechanical Engineering, Imperial College London（伦敦帝国理工学院机械工程系）
Kinematic analysis of a parallel robot for minimally invasive surgery
CESTER, Technical University of Cluj-Napoca（克卢日-纳波卡技术大学 CESTER）；École Centrale Nantes, Nantes Université, CNRS, LS2N, UMR 6004（南特中央理工大学、南特大学、法国国家科学研究中心、LS2N实验室、UMR 6004）；Technical Sciences Academy of Romania（罗马尼亚技术科学院）
Gaining spatial awareness of the Operating Room (OR) for surgical robotic systems is a key technology that can enable intelligent applications aiming at improved OR workflow. In this work, we present a method for semantic dense reconstruction of the OR scene using multiple RGB-D cameras attached and registered to the da Vinci Xi surgical system. We developed a novel SLAM approach for robot pose tracking in dynamic OR environments and dense reconstruction of the static OR table object. We validated our techniques in a mock OR by collecting data sequences with corresponding optical tracking traj…
University of British Columbia（不列颠哥伦比亚大学）；Technical University of Munich（慕尼黑工业大学）
Surgical instrument segmentation for robot-assisted surgery is needed for accurate instrument tracking and augmented reality overlays. Therefore, the topic has been the subject of a number of recent papers in the CAI community. Deep learning-based methods have shown state-of-the-art performance for surgical instrument segmentation, but their results depend on labelled data. However, labelled surgical data is of limited availability and is a bottleneck in surgical translation of these methods. In this paper, we demonstrate the limited generalizability of these methods on different datasets, inc…
State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统管理与控制国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
The real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, it is still a challenging task to implement deep learning models to do real-time segmentation for surgical instruments due to their high computational costs and slow inference speed. In this paper, we propose an attention-guided lightweight network (LWANet), which can segment surgical instruments in real-time. LWANet adopts encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2, and the decoder consists of depthwise separable convolution, attention fusion …
This work proves that semantic segmentation on minimally invasive surgical instruments can be improved by using training data that has been augmented through domain adaptation. The benefit of this method is twofold. Firstly, it suppresses the need of manually labeling thousands of images by transforming synthetic data into realistic-looking data. To achieve this, a CycleGAN model is used, which transforms a source dataset to approximate the domain distribution of a target dataset. Secondly, this newly generated data with perfect labels is utilized to train a semantic segmentation neural networ…
ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding
Centre for Artificial Intelligence and Robotics (CAIR), Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；Chinese University of Hong Kong（香港中文大学）
SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentation
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；School of Software Engineering, South China University of Technology（华南理工大学软件工程学院）；Department of Data Science & AI, Monash University（莫纳什大学数据科学与人工智能系）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机科学学院）；Department of Computer Science, University of Rochester（罗切斯特大学计算机科学系）
本文提出 SurgicalPart-SAM (SP-SAM)，通过协同提示、跨模态编码器和部分到整体自适应融合与分层解码，将器械结构知识与 SAM 集成，解决手术器械分割中结构复杂和提示不灵活的问题，以最少的可调参数实现了最先进的性能。
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院威康/EPSRC介入与外科科学中心）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
Semantic tool segmentation in surgical videos is important for surgical scene understanding and computer-assisted interventions as well as for the development of robotic automation. The problem is challenging because different illumination conditions, bleeding, smoke and occlusions can reduce algorithm robustness. At present labelled data for training deep learning models is still lacking for semantic surgical instrument segmentation and in this paper we show that it may be possible to use robot kinematic data coupled with laparoscopic images to alleviate the labelling problem. We propose a ne…
Manual labeling of gestures in robot-assisted surgery is labor intensive, prone to errors, and requires expertise or training. We propose a method for automated and explainable generation of gesture transcripts that leverages the abundance of data for image segmentation. Surgical context is detected using segmentation masks by examining the distances and intersections between the tools and objects. Next, context labels are translated into gesture transcripts using knowledge-based Finite State Machine (FSM) and data-driven Long Short Term Memory (LSTM) models. We evaluate the performance of eac…
BARNet: Bilinear Attention Network with Adaptive Receptive Fields for Surgical Instrument Segmentation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
ICube, University of Strasbourg（斯特拉斯堡大学ICube研究所）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Politecnico di Milano（米兰理工大学）
3-D pose estimation of instruments is a crucial step towards automatic scene understanding in robotic minimally invasive surgery. Although robotic systems can potentially directly provide joint values, this information is not commonly exploited inside the operating room, due to its possible unreliability, limited access and the time-consuming calibration required, especially for continuum robots. For this reason, standard approaches for 3-D pose estimation involve the use of external tracking systems. Recently, image-based methods have emerged as promising, non-invasive alternatives. While man…
Biomedical Image Analysis Group, Imperial College London（伦敦帝国理工学院生物医学图像分析组）；Dept. of Biomedical Engineering, National University of Singapore（新加坡国立大学生物医学工程系）；Dept. of Electronics and Communication Engineering, National Institute of Technology, Tiruchirappalli（蒂鲁吉拉帕利国家技术学院电子与通信工程系）；Dept. of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）
Global and local relational reasoning enable scene understanding models to perform human-like scene analysis and understanding. Scene understanding enables better semantic segmentation and object-to-object interaction detection. In the medical domain, a robust surgical scene understanding model allows the automation of surgical skill evaluation, real-time monitoring of surgeon's performance and post-surgical analysis. This paper introduces a globally-reasoned multi-task surgical scene understanding model capable of performing instrument segmentation and tool-tissue interaction detection. Here,…
Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglement
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Chair for Computer Aided Medical Procedures, Technical University of Munich（慕尼黑技术大学计算机辅助医疗程序主席职位）；Department of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程系）；Center for Computational and Molecular Biology, Brown University（布朗大学计算与分子生物学中心）
An analysis of higher-order kinematics formalisms for an innovative surgical parallel robot
Technical University of Cluj-Napoca（克卢日-纳波卡技术大学）；Gheorghe Asachi Technical University of Iasi（雅西格奥尔基·阿萨基技术大学）；Technical Sciences Academy of Romania（罗马尼亚技术科学院）；École Centrale Nantes（南特中央理工学院）；Nantes Université（南特大学）；CNRS（法国国家科学研究中心）；LS2N（南特数字科学实验室）
Self-supervised, multi-modal learning has been successful in holistic representation of complex scenarios. This can be useful to consolidate information from multiple modalities which have multiple, versatile uses. Its application in surgical robotics can lead to simultaneously developing a generalised machine understanding of the surgical process and reduce the dependency on quality, expert annotations which are generally difficult to obtain. We develop a self-supervised, multi-modal representation learning paradigm that learns representations for surgical gestures from video and kinematics. …
