In this study, we present an implementation strategy for a robot that performs peg transfer tasks in Fundamentals of Laparoscopic Surgery (FLS) via imitation learning, aimed at the development of an autonomous robot for laparoscopic surgery. Robotic laparoscopic surgery presents two main challenges: (1) the need to manipulate forceps using ports established on the body surface as fulcrums, and (2) difficulty in perceiving depth information when working with a monocular camera that displays its images on a monitor. Especially, regarding issue (2), most prior research has assumed the availability of depth images or models of a target to be operated on. Therefore, in this study, we achieve more accurate imitation learning with only monocular images by extracting motion constraints from one exemplary motion of skilled operators, collecting data based on these constraints, and conducting imitation learning based on the collected data. We implemented an overall system using two Franka Emika Panda Robot Arms and validated its effectiveness.
The Chinese University of Hong Kong（香港中文大学）；The Johns Hopkins University（约翰斯·霍普金斯大学）
Robot-assisted surgery has become progressively more and more popular due to its clinical advantages. In the meanwhile, the artificial intelligence and augmented reality in robotic surgery are developing rapidly and receive lots of attention. However, current methods have not discussed the coherent integration of AI and AR in robotic surgery. In this paper, we develop a novel system by seamlessly merging artificial intelligence module and augmented reality visualization to automatically generate the surgical guidance for robotic surgery education. Specifically, we first leverage reinforcement …
Johns Hopkins University（约翰斯·霍普金斯大学）；Children’s National Hospital（国立儿童医院）
Autonomous robotic surgery has the potential to provide efficacy, safety, and consistency independent of individual surgeons skill and experience. Autonomous soft-tissue surgery in unstructured and deformable environments is especially challenging as it necessitates intricate imaging, tissue tracking and surgical planning techniques, as well as a precise execution via highly adaptable control strategies. In the laparoscopic setting, soft-tissue surgery is even more challenging due to the need for high maneuverability and repeatability under motion and vision constraints. We demonstrate the fir…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Heidelberg University Hospital（海德堡大学医院）；Friedrich-Alexander-University Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Technical University Dresden（德累斯顿工业大学）
Recent advances in reinforcement learning (RL) have increased the promise of introducing cognitive assistance and automation to robot-assisted laparoscopic surgery (RALS). However, progress in algorithms and methods depends on the availability of standardized learning environments that represent skills relevant to RALS. We present LapGym, a framework for building RL environments for RALS that models the challenges posed by surgical tasks, and sofa_env, a diverse suite of 12 environments. Motivated by surgical training, these environments are organized into 4 tracks: Spatial Reasoning, Deformab…
University College London（伦敦大学学院）；Welcome/EPSRC Centre for Interventional and Surgical Sciences（惠康/工程与物理科学研究委员会介入与外科科学中心）
In robotic-assisted partial nephrectomy, surgeons remove a part of a kidney often due to the presence of a mass. A drop-in ultrasound probe paired to a surgical robot is deployed to execute multiple swipes over the kidney surface to localise the mass and define the margins of resection. This sub-task is challenging and must be performed by a highly skilled surgeon. Automating this sub-task may reduce cognitive load for the surgeon and improve patient outcomes. The overall goal of this work is to autonomously move the ultrasound probe on the surface of the kidney taking advantage of the use of …
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院威康/EPSRC介入与外科科学中心）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
Semantic tool segmentation in surgical videos is important for surgical scene understanding and computer-assisted interventions as well as for the development of robotic automation. The problem is challenging because different illumination conditions, bleeding, smoke and occlusions can reduce algorithm robustness. At present labelled data for training deep learning models is still lacking for semantic surgical instrument segmentation and in this paper we show that it may be possible to use robot kinematic data coupled with laparoscopic images to alleviate the labelling problem. We propose a ne…
LapSurgie: Humanoid Robots Performing Surgery via Teleoperated Handheld Laparoscopy
Department of Electrical and Computer Engineering, University of California San Diego（加州大学圣地亚哥分校电气与计算机工程系）；School of Electrical and Computer Engineering, Cornell University（康奈尔大学电气与计算机工程学院）；UC San Diego Health（加州大学圣地亚哥医学中心）
Learning-based Force Sensing and Impedance Matching for Safe Haptic Feedback in Robot-assisted Laparoscopic Surgery
organization= Surgical Robotics Laboratory (SRL), Mechanical Engineering Department, Concordia University , city= Montreal , state= QC , country= Canada；Robotics (SuPER) Centre, Department of Surgery, McGill University , city= Montreal , state= QC , country= Canada；Joyce Posluns Centre for Image Guided Innovation \& Therapeutic Intervention (PCIGITI), The Hospital for Sick Children (SickKids) , city= Toronto , state= ON , country= Canada
SLAM assisted 3D tracking system for laparoscopic surgery
United Imaging Research Institute of Intelligent Imaging（联影智能影像研究院）；University of Michigan（密歇根大学）；United Imaging（联影医疗）；United Imaging of Surgery（联影外科研究院）
Workspace Analysis for Laparoscopic Rectal Surgery : A Preliminary Study
Centre Hospitalier Universitaire（大学中心医院）；CSIR-Central Scientific Instruments Organisation（CSIR中央科学仪器组织）；Academy of Scientific and Innovative Research (AcSIR)（科学与创新研究院）；École Centrale Nantes（南特中央理工学院）；Nantes Université（南特大学）；CNRS（法国国家科学研究中心）；Research Center for Industrial Robots Simulation and Testing（工业机器人仿真与测试研究中心）；Technical University of Cluj-Napoca（克卢日-纳波卡技术大学）
University of Strasbourg（斯特拉斯堡大学）；Dedan Kimathi University of Technology（德丹·基马西理工大学）；ICube UMR 7357 Robotics Department University of Strasbourg（ICube UMR 7357 机器人系，斯特拉斯堡大学）；ICube UMR 7357 University of Strasbourg, Centre National de la Recherche Scientifique (CNRS)（ICube UMR 7357，斯特拉斯堡大学，法国国家科学研究中心）
This paper builds on our previous work by exploiting Artificial Intelligence to predict individual grip force variability in manual robot control. Grip forces were recorded from various loci in the dominant and non dominant hands of individuals by means of wearable wireless sensor technology. Statistical analyses bring to the fore skill specific temporal variations in thousands of grip forces of a complete novice and a highly proficient expert in manual robot control. A brain inspired neural network model that uses the output metric of a Self Organizing Map with unsupervised winner take all le…
An Open-Source Robotics Research Platform for Autonomous Laparoscopic Surgery
Department of Translational Surgical Oncology, NCT/UCC Dresden, a partnership between DKFZ, MF UKD, TUD, HZDR Dresden, Germany（转化外科肿瘤学系，NCT/UCC德累斯顿，DKFZ、MF UKD、TUD、HZDR德累斯顿的联合体）；Cluster of Excellence-CeTI, TUD, Germany（卓越中心- CeTI，TUD，德国）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Germany（visceral、胸腔和血管外科系，医学学院和卡尔·格奥尔格·卡尔医院，TUD，德国）
Point Cloud Segmentation for Autonomous Clip Positioning in Laparoscopic Cholecystectomy on a Phantom
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；HIDSS4Health - Helmholtz Information and Data Science School for Health（亥姆霍兹信息与数据科学健康学校）；Friedrich-Alexander-University, Erlangen-Nuremberg（弗里德里希-亚历山大大学埃尔朗根-纽伦堡）；University Hospital Carl Gustav Carus and Centre for Tactile Internet with Human-in-the-loop (CeTI), Dresden University of Technology（卡尔·古斯塔夫·卡鲁斯大学医院及德累斯顿工业大学触觉互联网人机共融卓越中心）
A novel Integrated Motion Tracking Device (IMTD) for Objective Laparoscopic Training Assessment: Development and Validation
Pprime Institute CNRS（法国国家科学研究中心Pprime研究所）；ENSMA（国立高等机械与航空技术学院）；University of Poitiers（普瓦捷大学）；Mechanical Laboratory of Sousse (LMS)（苏塞机械实验室）；National Engineering School of Sousse（苏塞国立工程师学院）；University of Sousse（苏塞大学）；National Engineering School of Monastir（莫纳斯提尔国立工程师学院）；University of Monastir（莫纳斯提尔大学）；Computer Engineering, Production and Maintenance Laboratory (LGIPM)（计算机工程、生产与维护实验室）；University of Lorraine（洛林大学）
Contact-Stable Deformable Tissue Simulation Using Implicit Integration and Live-Pose Grasp Constraints for Laparoscopic Surgery Robot Policy Evaluation
Seoul National University Hospital（首尔大学医院）；Rosota Inc.（Rosota公司）；Seoul National University College of Medicine（首尔大学医学院）；Seoul National University（首尔大学）；Institute of Convergence Medicine with Innovative Technology, Seoul National University Hospital（首尔大学医院创新技术融合医学研究所）；Eulji University College of Medicine（乙支大学医学院）；Chungang University（中央大学）
University of California, Berkeley（加州大学伯克利分校）；SRI International（SRI国际）
Peg transfer is a well-known surgical training task in the Fundamentals of Laparoscopic Surgery (FLS). While human sur-geons teleoperate robots such as the da Vinci to perform this task with high speed and accuracy, it is challenging to automate. This paper presents a novel system and control method using a da Vinci Research Kit (dVRK) surgical robot and a Zivid depth sensor, and a human subjects study comparing performance on three variants of the peg-transfer task: unilateral, bilateral without handovers, and bilateral with handovers. The system combines 3D printing, depth sensing, and deep …
Grenoble University Hospital（格勒诺布尔大学医院）；Saint-Etienne University Hospital（圣埃蒂安大学医院）；TIMC-IMAG/GMCAO Lab (CNRS 5525)（TIMC-IMAG/GMCAO实验室（CNRS 5525））；Clinical Investigation Center, Grenoble University Hospital（格勒诺布尔大学医院临床研究中心）
Purpose: To report the feasibility and the safety of a surgeon-controlled robotic endoscope holder in laparoscopic surgery. Materials and methods: From March 2010 to September 2010, 20 patients were enrolled prospectively to undergo a laparoscopic surgery using an innovative robotic endoscope holder. Two surgeons performed 6 adrenalectomies, 4 sacrocolpopexies, 5 pyeloplasties, 4 radical prostatectomies and 1 radical nephrectomy. Demographic data, overall set-up time, operative time, number of assistants needed were reviewed. Surgeon's satisfaction regarding the ergonomics was assessed using a…
Semi-Autonomous Laparoscopic Robot Docking with Learned Hand-Eye Information Fusion
School of Biomedical Engineering & Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与影像科学学院）；School of Mechatronical Engineering, Beijing Institute of Technology（北京理工大学机电学院）；School of Medical Technology, Beijing Institute of Technology（北京理工大学医学技术学院）
Digital Surgery Limited（数字手术有限公司）；University College London（伦敦大学学院）；Kasetsart University（泰国农业大学）
Robotic surgery and novel surgical instrumentation present great potentials towards safer, more accurate and consistent minimally invasive surgery. However, their adoption is dependent to the access to training facilities and extensive surgical training. Robotic instruments require different dexterity skills compared to open or laparoscopic. Surgeons, therefore, are required to invest significant time by attending extensive training programs. Contrary, hands on experiences represent an additional operational cost for hospitals as the availability of robotic systems for training purposes is lim…
University of California, Berkeley（加利福尼亚大学伯克利分校）；UC San Francisco East Bay（加州大学旧金山分校东湾校区）；SRI International（斯坦福国际研究院）
Recent advances in depth-sensing have significantly increased accuracy, resolution, and frame rate, as shown in the 1920x1200 resolution and 13 frames per second Zivid RGBD camera. In this study, we explore the potential of depth sensing for efficient and reliable automation of surgical subtasks. We consider a monochrome (all red) version of the peg transfer task from the Fundamentals of Laparoscopic Surgery training suite implemented with the da Vinci Research Kit (dVRK). We use calibration techniques that allow the imprecise, cable-driven da Vinci to reduce error from 4-5 mm to 1-2 mm in the…
The surgical paradigm of Minimally Invasive Surgery (MIS) has been a key driver to the adoption of robotic surgical assistance. Progress in the last three decades has led to a gradual transition from manual laparoscopic surgery with rigid instruments to robot-assisted surgery. In the last decade, the increasing demand for new surgical paradigms to enable access into the anatomy without skin incision (intraluminal surgery) or with a single skin incision (Single Port Access surgery - SPA) has led researchers to investigate snake-like flexible surgical devices. In this chapter, we first present a…
This study tries to explain the connection between communication modalities and levels of supervision in teleoperation during a dexterous task, like surgery. This concept is applied to two surgical related tasks: incision and peg transfer. It was found that as the complexity of the task escalates, the combination linking human supervision with a more expressive modality shows better performance than other combinations of modalities and control. More specifically, in the peg transfer task, the combination of speech modality and action level supervision achieves shorter task completion time (77.…
The use of robots in minimally invasive surgery has improved the quality of standard surgical procedures. So far, only the automation of simple surgical actions has been investigated by researchers, while the execution of structured tasks requiring reasoning on the environment and the choice among multiple actions is still managed by human surgeons. In this paper, we propose a framework to implement surgical task automation. The framework consists of a task-level reasoning module based on answer set programming, a low-level motion planning module based on dynamic movement primitives, and a sit…
MiniTac: An Ultra-Compact 8 mm Vision-Based Tactile Sensor for Enhanced Palpation in Robot-Assisted Minimally Invasive Surgery
State Key Laboratory of General Artificial Intelligence（国家关键人工智能实验室）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Institute for Artificial Intelligence（人工智能学院）；Shenyang Institute of Automation（沈阳自动化研究所）；Chinese Academy of Sciences（中国科学院）；PLA General Hospital（中国人民解放军总医院）
PURPOSE OF REVIEW: Robot-assisted laparoscopic surgery in urology has gained immense popularity with the daVinci system, but a lot of research teams are working on new robots. The purpose of this study is to review current urologic robots and present future development directions. RECENT FINDINGS: Future systems are expected to advance in two directions: improvements of remote manipulation robots and developments of image-guided robots. SUMMARY: The final goal of robots is to allow safer and more homogeneous outcomes with less variability of surgeon performance, as well as new tools to perform…
MVP-Tac: A Miniaturized Dual-Modal Vision and Photoelastic Tactile Sensor for Robot-Assisted Minimally Invasive Surgery
Elmore Family School of Electrical and Computer Engineering, Purdue University（普渡大学埃尔莫尔电气与计算机工程学院）；Edwardson School of Industrial Engineering, Purdue University（普渡大学爱德华森工业工程学院）；School of Mechanical Engineering, Purdue University（普渡大学机械工程学院）；Robotics Engineering Technology, Purdue University（普渡大学机器人工程技术学院）；Department of Civil Engineering, Stony Brook University（石溪大学土木工程系）；Department of Mechanical Engineering, Michigan State University（密歇根州立大学机械工程系）
The Chinese University of Hong Kong（香港中文大学）；T stone Robotics Institute（T石机器人研究所）；Department of Computer Science and Engineering（计算机科学与工程学系）
Laparoscopic Field of View (FOV) control is one of the most fundamental and important components in Minimally Invasive Surgery (MIS), nevertheless, the traditional manual holding paradigm may easily bring fatigue to surgical assistants, and misunderstanding between surgeons also hinders assistants to provide a high-quality FOV. Targeting this problem, we here present a data-driven framework to realize an automated laparoscopic optimal FOV control. To achieve this goal, we offline learn a motion strategy of laparoscope relative to the surgeon's hand-held surgical tool from our in-house surgical…
Bridging the Ex-Vivo to In-Vivo Gap: Synthetic Priors for Monocular Depth Estimation in Specular Surgical Environments
RoCAL, Rochester Institute of Technology（罗切斯特理工学院RoCAL实验室）；Department of Otolaryngology Head and Neck Surgery, University of Rochester Medical Center（罗切斯特大学医学中心耳鼻喉科与头颈外科部门）；University of Washington, Department of Otolaryngology–Head and Neck Surgery（华盛顿大学耳鼻喉科与头颈外科部门）
