Learning meaningful visual representations in an embedding space can facilitate generalization in downstream tasks such as action segmentation and imitation. In this paper, we learn a motion-centric representation of surgical video demonstrations by grouping them into action segments/sub-goals/options in a semi-supervised manner. We present Motion2Vec, an algorithm that learns a deep embedding feature space from video observations by minimizing a metric learning loss in a Siamese network: images from the same action segment are pulled together while pushed away from randomly sampled images of other segments, while respecting the temporal ordering of the images. The embeddings are iteratively segmented with a recurrent neural network for a given parametrization of the embedding space after pre-training the Siamese network. We only use a small set of labeled video segments to semantically align the embedding space and assign pseudo-labels to the remaining unlabeled data by inference on the learned model parameters. We demonstrate the use of this representation to imitate surgical suturing motions from publicly available videos of the JIGSAWS dataset. Results give 85.5 % segmentation accuracy on average suggesting performance improvement over several state-of-the-art baselines, while kinematic pose imitation gives 0.94 centimeter error in position per observation on the test set. Videos, code and data are available at https://sites.google.com/view/motion2vec
Self-supervised, multi-modal learning has been successful in holistic representation of complex scenarios. This can be useful to consolidate information from multiple modalities which have multiple, versatile uses. Its application in surgical robotics can lead to simultaneously developing a generalised machine understanding of the surgical process and reduce the dependency on quality, expert annotations which are generally difficult to obtain. We develop a self-supervised, multi-modal representation learning paradigm that learns representations for surgical gestures from video and kinematics. …
UCL（伦敦大学学院）；Università di Verona（维罗纳大学）；Politecnico di Milano（米兰理工大学）
Kinematic trajectories recorded from surgical robots contain information about surgical gestures and potentially encode cues about surgeon's skill levels. Automatic segmentation of these trajectories into meaningful action units could help to develop new metrics for surgical skill assessment as well as to simplify surgical automation. State-of-the-art methods for action recognition relied on manual labelling of large datasets, which is time consuming and error prone. Unsupervised methods have been developed to overcome these limitations. However, they often rely on tedious parameter tuning and…
ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube实验室、法国国家科学研究中心、斯特拉斯堡大学医院研究所）
Surgical tool segmentation in endoscopic images is the first step towards pose estimation and (sub-)task automation in challenging minimally invasive surgical operations. While many approaches in the literature have shown great results using modern machine learning methods such as convolutional neural networks, the main bottleneck lies in the acquisition of a large number of manually-annotated images for efficient learning. This is especially true in surgical context, where patient-to-patient differences impede the overall generalizability. In order to cope with this lack of annotated data, we…
Surgical Task Automation Using Actor-Critic Frameworks and Self-Supervised Imitation Learning
University of Edinburgh（爱丁堡大学）；Institute for Imaging, Data and Communications（成像、数据与通信研究所）；School of Engineering（工程学院）；TECNALIA（泰克纳利亚）；University of Strathclyde（思克莱德大学）；Centre for Precision Manufacturing（精密制造中心）；Department of Biomedical Engineering（生物医学工程系）
The Hamlyn Centre for Robotic Surgery, IGHI, Imperial College London（伦敦帝国理工学院哈姆林机器人手术中心）；Dyson Robotics Laboratory, Imperial College London（伦敦帝国理工学院戴森机器人实验室）；OpenAI；Department of Surgery and Cancer, Imperial College London（伦敦帝国理工学院外科与癌症系）
Robotic surgery has become a powerful tool for performing minimally invasive procedures, providing advantages in dexterity, precision, and 3D vision, over traditional surgery. One popular robotic system is the da Vinci surgical platform, which allows preoperative information to be incorporated into live procedures using Augmented Reality (AR). Scene depth estimation is a prerequisite for AR, as accurate registration requires 3D correspondences between preoperative and intraoperative organ models. In the past decade, there has been much progress on depth estimation for surgical scenes, such as …
Supervised Mixture-of-Experts for Surgical Grasping and Retraction
Department of Translational Surgical Oncology, NCT/UCC Dresden, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Dresden, Germany（translational Surgical Oncology部门，NCT/UCC Dresden，医学院和Carl Gustav Carus大学医院，TUD，德累斯顿，德国）；National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, and HZDR, Dresden, Germany（肿瘤疾病国家中心（NCT），NCT/UCC德累斯顿，DKFZ、医学院和Carl Gustav Carus大学医院、TUD以及HZDR之间的合作，德累斯顿，德国）；Faculty of Computer Science, TUD, Dresden, Germany（计算机科学系，TUD，德累斯顿，德国）；The Center for Tactile Internet with Human-in-the-Loop (CeTI), TUD, Dresden, Germany（带有Human-in-the-Loop的触觉互联网中心（CeTI），TUD，德累斯顿，德国）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Dresden, Germany（visceral、胸腔和血管外科部门，医学院和Carl Gustav Carus大学医院，TUD，德累斯顿，德国）
NUS Graduate School for Integrative Sciences and Engineering (NGS)（新加坡国立大学综合科学与工程研究生院（NGS））；National University of Singapore（新加坡国立大学）；University of Moratuwa（莫拉图瓦大学）；NIT Trichy（印度国立理工学院蒂鲁吉拉伯利分校）
Robot-assisted surgery is an emerging technology which has undergone rapid growth with the development of robotics and imaging systems. Innovations in vision, haptics and accurate movements of robot arms have enabled surgeons to perform precise minimally invasive surgeries. Real-time semantic segmentation of the robotic instruments and tissues is a crucial step in robot-assisted surgery. Accurate and efficient segmentation of the surgical scene not only aids in the identification and tracking of instruments but also provided contextual information about the different tissues and instruments be…
Imperial College London（伦敦帝国理工学院）；University of Bristol（布里斯托尔大学）
In recent decades, the tremendous benefits surgical robots have brought to surgeons and patients have been witnessed. With the dexterous operation and the great precision, surgical robots can offer patients less recovery time and less hospital stay. However, the controls for current surgical robots in practical usage are fully carried out by surgeons via teleoperation. During the surgery process, there exists a lot of repetitive but simple manipulation, which can cause unnecessary fatigue to the surgeons. In this paper, we proposed a deep reinforcement learning-based semi-autonomous control fr…
SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy
Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Department of Thoracic Surgery, Peking University People’s Hospital（北京大学人民医院胸外科）；Thoracic Oncology Institute, Peking University People’s Hospital（北京大学人民医院胸部肿瘤研究所）；Research Unit of Intelligence Diagnosis and Treatment in Early Non-small Cell Lung Cancer, Chinese Academy of Medical Sciences（中国医学科学院早期非小细胞肺癌智能诊断与治疗研究组）；Institute of Advanced Clinical Medicine, Peking University（北京大学先进临床医学院）；Beijing Key Laboratory of Innovative Application of Big Data in Lung Cancer, Peking University People’s Hospital（北京大学人民医院肺癌大数据创新应用北京市重点实验室）
Uncertainty-aware Self-supervised Learning for Cross-domain Technical Skill Assessment in Robot-assisted Surgery
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Scuola Superiore Sant’Anna（圣安娜高等研究院）；Politecnico di Milano（米兰理工大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
ICube, University of Strasbourg（斯特拉斯堡大学ICube研究所）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Politecnico di Milano（米兰理工大学）
3-D pose estimation of instruments is a crucial step towards automatic scene understanding in robotic minimally invasive surgery. Although robotic systems can potentially directly provide joint values, this information is not commonly exploited inside the operating room, due to its possible unreliability, limited access and the time-consuming calibration required, especially for continuum robots. For this reason, standard approaches for 3-D pose estimation involve the use of external tracking systems. Recently, image-based methods have emerged as promising, non-invasive alternatives. While man…
SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis
National University of Singapore（国立新加坡大学）；Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR)（生物信息研究所（BII），科技研究局（A*STAR））；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and the Department of Medical Physics and Biomedical Engineering, University College London（Wellcome/EPSRC介入与外科科学中心（WEISS）及伦敦大学学院医学物理与生物医学工程系）
