To improve the efficiency of surgical trajectory segmentation for robot learning in robot-assisted minimally invasive surgery, this paper presents a fast unsupervised method using video and kinematic data, followed by a promoting procedure to address the over-segmentation issue. Unsupervised deep learning network, stacking convolutional auto-encoder, is employed to extract more discriminative features from videos in an effective way. To further improve the accuracy of segmentation, on one hand, wavelet transform is used to filter out the noises existed in the features from video and kinematic data. On the other hand, the segmentation result is promoted by identifying the adjacent segments with no state transition based on the predefined similarity measurements. Extensive experiments on a public dataset JIGSAWS show that our method achieves much higher accuracy of segmentation than state-of-the-art methods in the shorter time.
Learning-based Hierarchical Tracheal Anatomy Understanding from Sparse Surgical Demonstration Annotations for Ultrasound Robots
The Chinese University of Hong Kong（香港中文大学）；Imperial College London（伦敦帝国学院）；Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）；SLAI Shenzhen Loop Area Institute（深圳环域先进人工智能研究院）
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）
Surgical gesture recognition is important for surgical data science and computer-aided intervention. Even with robotic kinematic information, automatically segmenting surgical steps presents numerous challenges because surgical demonstrations are characterized by high variability in style, duration and order of actions. In order to extract discriminative features from the kinematic signals and boost recognition accuracy, we propose a multi-task recurrent neural network for simultaneous recognition of surgical gestures and estimation of a novel formulation of surgical task progress. To show the…
SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splatting
The Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学）；Shenzhen Research Institute, CUHK, Shenzhen, China（深圳研究学院）；Technical University of Munich, Munich, Germany（慕尼黑技术大学）；University of Strasbourg & IHU Strasbourg, Strasbourg, France（斯特拉斯堡大学及斯特拉斯堡IHU）；University College London, London, United Kingdom（伦敦大学学院）
Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling
NVIDIA；The Chinese University of Hong Kong（香港中文大学）；Sung Kyun Kwan University（全州大学）；Wenzhou Medical University（温州医学院）；National University of Singapore（新加坡国立大学）；Ruijin Hospital（瑞金医院）
DARIL: When Imitation Learning outperforms Reinforcement Learning in Surgical Action Planning
Surgical and Interventional Engineering, King's College London, London, UK（外科与介入工程，伦敦国王学院，伦敦，英国）；Interventional Engineering, King's College London, London, UK（介入工程，伦敦国王学院，伦敦，英国）
该研究在 CholecT50 上比较手术动作规划中的 IL 与 RL，提出 DARIL 基线，发现其显著优于三种 RL 方法，并揭示专家测试分布对 IL 的系统性偏好。
Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Northwestern University（西北大学）；University of Alberta（阿尔伯塔大学）；Yale University（耶鲁大学）；Nanfang Hospital（南华医院）；Shenzhen University of Advanced Technology（深圳大学先进技术研究院）
Despite significant developments in the design of surgical robots and automated techniques for objective evaluation of surgical skills, there are still challenges in ensuring safety in robot-assisted minimally-invasive surgery (RMIS). This paper presents a runtime monitoring system for the detection of executional errors during surgical tasks through the analysis of kinematic data. The proposed system incorporates dual Siamese neural networks and knowledge of surgical context, including surgical tasks and gestures, their distributional similarities, and common error modes, to learn the differe…
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University College London（伦敦大学学院）
Surgical instrument segmentation -- in general a pixel classification task -- is fundamentally crucial for promoting cognitive intelligence in robot-assisted surgery (RAS). However, previous methods are struggling with discriminating instrument types and instances. To address the above issues, we explore a mask classification paradigm that produces per-segment predictions. We propose TraSeTR, a novel Track-to-Segment Transformer that wisely exploits tracking cues to assist surgical instrument segmentation. TraSeTR jointly reasons about the instrument type, location, and identity with instance-…
University of California, Berkeley（加州大学伯克利分校）；Google Brain（谷歌大脑）；Intel AI Labs（英特尔人工智能实验室）
Learning meaningful visual representations in an embedding space can facilitate generalization in downstream tasks such as action segmentation and imitation. In this paper, we learn a motion-centric representation of surgical video demonstrations by grouping them into action segments/sub-goals/options in a semi-supervised manner. We present Motion2Vec, an algorithm that learns a deep embedding feature space from video observations by minimizing a metric learning loss in a Siamese network: images from the same action segment are pulled together while pushed away from randomly sampled images of …
Localising under the drape: proprioception in the era of distributed surgical robotic system
School of Biomedical Engineering & Imaging Sciences, King’s College London（国王学院伦敦生物医学工程与成像科学学院）；Research in Orthopaedic Computer Science, Balgrist University Hospital（骨科计算机科学研究，巴尔格里斯特大学医院）；Faculty of Engineering Technology, KU Leuven（工程科技学院，鲁汶大学）；Noah’s Ark Lab, Huawei（华为诺亚实验室）；University Spine Center Zurich, Balgrist University Hospital, University of Zurich（苏黎世大学脊柱中心，巴尔格里斯特大学医院，苏黎世大学）；SpineGuard, Paris, France（脊柱守护，法国巴黎）；Institut Des Systèmes Intelligents et de Robotique, Sorbonne University（智能系统与机器人研究所，索邦大学）
Hamlyn Centre for Robotic Surgery, Imperial College London（伦敦帝国理工学院哈姆林机器人手术中心）；Institute of Image Processing & Pattern Recognition, Shanghai Jiao Tong University（上海交通大学图像处理与模式识别研究所）
In robotic surgery, task automation and learning from demonstration combined with human supervision is an emerging trend for many new surgical robot platforms. One such task is automated anastomosis, which requires bimanual needle handling and suture detection. Due to the complexity of the surgical environment and varying patient anatomies, reliable suture detection is difficult, which is further complicated by occlusion and thread topologies. In this paper, we propose a multi-stage framework for suture thread detection based on deep learning. Fully convolutional neural networks are used to ob…
Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model. To this end, we propose Neural Task Graph (NTG) Networks, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy. We empirically show NTG achieves inter-task generalization on two complex…
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）
Objective: Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classif…
Anomaly detection in Minimally-Invasive Surgery (MIS) traditionally requires a human expert monitoring the procedure from a console. Data scarcity, on the other hand, hinders what would be a desirable migration towards autonomous robotic-assisted surgical systems. Automated anomaly detection systems in this area typically rely on classical supervised learning. Anomalous events in a surgical setting, however, are rare, making it difficult to capture data to train a detection model in a supervised fashion. In this work we thus propose an unsupervised approach to anomaly detection for robotic-ass…
AUTOLAB at the University of California, Berkeley（加州大学伯克利分校AUTOLAB）；Honda Research Institute USA, Inc.（美国本田研究所）
Robotic fabric manipulation has applications in home robotics, textiles, senior care and surgery. Existing fabric manipulation techniques, however, are designed for specific tasks, making it difficult to generalize across different but related tasks. We extend the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different fabric manipulation tasks with a single goal-conditioned policy. We introduce VisuoSpatial Foresight (VSF), which builds on prior work by learning visual dynamics on domain randomized RGB images and depth maps simultaneously and…
