A Learning-Driven Framework with Spatial Optimization For Surgical Suture Thread Reconstruction and Autonomous Grasping Under Multiple Topologies and Environmental Noises
The Chinese University of Hong Kong（香港中文大学）；Imperial College London（伦敦帝国学院）
Bo Lu, Wei Chen, Yue-Ming Jin, Dandan Zhang, Qi Dou, Henry K. Chu, Pheng-Ann Heng, Yun-Hui Liu
Surgical knot tying is one of the most fundamental and important procedures in surgery, and a high-quality knot can significantly benefit the postoperative recovery of the patient. However, a longtime operation may easily cause fatigue to surgeons, especially during the tedious wound closure task. In this paper, we present a vision-based method to automate the suture thread grasping, which is a sub-task in surgical knot tying and an intermediate step between the stitching and looping manipulations. To achieve this goal, the acquisition of a suture's three-dimensional (3D) information is critical. Towards this objective, we adopt a transfer-learning strategy first to fine-tune a pre-trained model by learning the information from large legacy surgical data and images obtained by the on-site equipment. Thus, a robust suture segmentation can be achieved regardless of inherent environment noises. We further leverage a searching strategy with termination policies for a suture's sequence inference based on the analysis of multiple topologies. Exact results of the pixel-level sequence along a suture can be obtained, and they can be further applied for a 3D shape reconstruction using our optimized shortest path approach. The grasping point considering the suturing criterion can be ultimately acquired. Experiments regarding the suture 2D segmentation and ordering sequence inference under environmental noises were extensively evaluated. Results related to the automated grasping operation were demonstrated by simulations in V-REP and by robot experiments using Universal Robot (UR) together with the da Vinci Research Kit (dVRK) adopting our learning-driven framework.
University of California San Diego（加州大学圣地亚哥分校）；University of New Mexico（新墨西哥大学）
Automating the process of manipulating and delivering sutures during robotic surgery is a prominent problem at the frontier of surgical robotics, as automating this task can significantly reduce surgeons' fatigue during tele-operated surgery and allow them to spend more time addressing higher-level clinical decision making. Accomplishing autonomous suturing and suture manipulation in the real world requires accurate suture thread localization and reconstruction, the process of creating a 3D shape representation of suture thread from 2D stereo camera surgical image pairs. This is a very challen…
Autonomous suturing has been a long-sought-after goal for surgical robotics. Outside of staged environments, accurate localization of suture needles is a critical foundation for automating various suture needle manipulation tasks in the real world. When localizing a needle held by a gripper, previous work usually tracks them separately without considering their relationship. Because of the significant errors that can arise in the stereo-triangulation of objects and instruments, their reconstructions may often not be consistent. This can lead to unrealistic tool-needle grasp reconstructions tha…
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland, College Park（马里兰大学帕克分校）；Weinberg Medical Physics, Inc.（温伯格医学物理公司）
Real-time visual localization of needles is necessary for various surgical applications, including surgical automation and visual feedback. In this study we investigate localization and autonomous robotic control of needles in the context of our magneto-suturing system. Our system holds the potential for surgical manipulation with the benefit of minimal invasiveness and reduced patient side effects. However, the non-linear magnetic fields produce unintuitive forces and demand delicate position-based control that exceeds the capabilities of direct human manipulation. This makes automatic needle…
Politecnico di Milano（米兰理工大学）；University College London（伦敦大学学院）
Robot-assisted laparoscopic prostatectomy (RALP) is a treatment for prostate cancer that involves complete or nerve sparing removal prostate tissue that contains cancer. After removal the bladder neck is successively sutured directly with the urethra. The procedure is called urethrovesical anastomosis and is one of the most dexterity demanding tasks during RALP. Two suturing instruments and a pair of needles are used in combination to perform a running stitch during urethrovesical anastomosis. While robotic instruments provide enhanced dexterity to perform the anastomosis, it is still highly c…
In recent years, surgical robots have become instrumental tools for assisting surgeons in performing complex surgical procedures in hospitals. Unlike conventional surgical methods, robotic systems help surgeons, for example, to perform minimally invasive surgical procedures while enhancing the precision and control of operations (e.g. tiny incisions, wound sutures, endoscopic suturing, among others). To this extent, it is essential to consider several factors that may influence the feasibility and decision making of employing robotic systems in surgical procedures. In this paper, we propose an…
Johns Hopkins University（约翰斯·霍普金斯大学）；Children’s National Hospital（国立儿童医院）
Autonomous robotic surgery has the potential to provide efficacy, safety, and consistency independent of individual surgeons skill and experience. Autonomous soft-tissue surgery in unstructured and deformable environments is especially challenging as it necessitates intricate imaging, tissue tracking and surgical planning techniques, as well as a precise execution via highly adaptable control strategies. In the laparoscopic setting, soft-tissue surgery is even more challenging due to the need for high maneuverability and repeatability under motion and vision constraints. We demonstrate the fir…
University of California at Los Angeles（加利福尼亚大学洛杉矶分校）；David Geffen School of Medicine（大卫·格芬医学院）
Needle shape, diameter, and path are critical parameters that directly affect suture depth and tissue trauma in autonomous suturing. This paper presents an optimization-based approach to specify these parameters. Given clinical suturing guidelines, a kinematic model of needle-tissue interaction was developed to quantify suture parameters and constraints. The model was further used to formulate constant curvature needle path planning as a nonlinear optimization problem. The optimization results were confirmed experimentally with the Raven II surgical system. The proposed needle path planning al…
Supervised Mixture-of-Experts for Surgical Grasping and Retraction
Department of Translational Surgical Oncology, NCT/UCC Dresden, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Dresden, Germany（translational Surgical Oncology部门，NCT/UCC Dresden，医学院和Carl Gustav Carus大学医院，TUD，德累斯顿，德国）；National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, and HZDR, Dresden, Germany（肿瘤疾病国家中心（NCT），NCT/UCC德累斯顿，DKFZ、医学院和Carl Gustav Carus大学医院、TUD以及HZDR之间的合作，德累斯顿，德国）；Faculty of Computer Science, TUD, Dresden, Germany（计算机科学系，TUD，德累斯顿，德国）；The Center for Tactile Internet with Human-in-the-Loop (CeTI), TUD, Dresden, Germany（带有Human-in-the-Loop的触觉互联网中心（CeTI），TUD，德累斯顿，德国）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Dresden, Germany（visceral、胸腔和血管外科部门，医学院和Carl Gustav Carus大学医院，TUD，德累斯顿，德国）
Reeling It In: Flexible Needle Pick Up via Thread Manipulation for Autonomous Suturing
University of California San Diego（加利福尼亚大学圣迭戈分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；Carnegie Mellon University（卡内基梅隆大学）；UC San Diego Health（加州大学圣迭戈分校健康中心）
RoboNurse-VLA: Robotic Scrub Nurse System based on Vision-Language-Action Model
Multi-Scale Medical Robotics Centre, Ltd., The Chinese University of Hong Kong（香港中文大学多尺度医疗机器人中心有限公司）；Istituto Italiano di Tecnologia（意大利技术研究院）；Department of Surgery, The Chinese University of Hong Kong（香港中文大学外科系）
An Open-Source Robotics Research Platform for Autonomous Laparoscopic Surgery
Department of Translational Surgical Oncology, NCT/UCC Dresden, a partnership between DKFZ, MF UKD, TUD, HZDR Dresden, Germany（转化外科肿瘤学系，NCT/UCC德累斯顿，DKFZ、MF UKD、TUD、HZDR德累斯顿的联合体）；Cluster of Excellence-CeTI, TUD, Germany（卓越中心- CeTI，TUD，德国）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Germany（visceral、胸腔和血管外科系，医学学院和卡尔·格奥尔格·卡尔医院，TUD，德国）
