Recent advances in depth-sensing have significantly increased accuracy, resolution, and frame rate, as shown in the 1920x1200 resolution and 13 frames per second Zivid RGBD camera. In this study, we explore the potential of depth sensing for efficient and reliable automation of surgical subtasks. We consider a monochrome (all red) version of the peg transfer task from the Fundamentals of Laparoscopic Surgery training suite implemented with the da Vinci Research Kit (dVRK). We use calibration techniques that allow the imprecise, cable-driven da Vinci to reduce error from 4-5 mm to 1-2 mm in the task space. We report experimental results for a handover-free version of the peg transfer task, performing 20 and 5 physical episodes with single- and bilateral-arm setups, respectively. Results over 236 and 49 total block transfer attempts for the single- and bilateral-arm peg transfer cases suggest that reliability can be attained with 86.9 % and 78.0 % for each individual block, with respective block transfer speeds of 10.02 and 5.72 seconds. Supplementary material is available at https://sites.google.com/view/peg-transfer.
Imperial College London（伦敦帝国理工学院）；University of Leeds（利兹大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Worcester Polytechnic Institute（伍斯特理工学院）；Intuitive Surgical Inc（直觉外科公司）
Robotic-assisted surgery is now well-established in clinical practice and has become the gold standard clinical treatment option for several clinical indications. The field of robotic-assisted surgery is expected to grow substantially in the next decade with a range of new robotic devices emerging to address unmet clinical needs across different specialities. A vibrant surgical robotics research community is pivotal for conceptualizing such new systems as well as for developing and training the engineers and scientists to translate them into practice. The da Vinci Research Kit (dVRK), an acade…
On-the-fly hand-eye calibration for the da Vinci surgical robot
Department of Mechanical Engineering, Imperial College London（帝国理工学院机械工程系）；Mechatronics in Medicine Laboratory（医学机电实验室）；Hamlyn Centre for Robotics Surgery（机器人外科哈姆林中心）
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Kyung Hee University（庆熙大学）
Innovations from surgical robotic research rarely translates to live surgery due to the significant difference between the lab and a live environment. Live environments require considerations that are often overlooked during early stages of research such as surgical staff, surgical procedure, and the challenges of working with live tissue. One such example is the da Vinci Research Kit (dVRK) which is used by over 40 robotics research groups and represents an open-sourced version of the da Vinci Surgical System. Despite dVRK being available for nearly a decade and the ideal candidate for transl…
Learning Autonomous Surgical Irrigation and Suction with the da Vinci Research Kit Using Reinforcement Learning
Department of Electrical and Computer Engineering, University of Alberta（电气与计算机工程系，阿尔伯塔大学）；Department of Biomedical Engineering, University of Alberta（生物医学工程系，阿尔伯塔大学）
University of California, Berkeley（加州大学伯克利分校）；SRI International（SRI国际）
Peg transfer is a well-known surgical training task in the Fundamentals of Laparoscopic Surgery (FLS). While human sur-geons teleoperate robots such as the da Vinci to perform this task with high speed and accuracy, it is challenging to automate. This paper presents a novel system and control method using a da Vinci Research Kit (dVRK) surgical robot and a Zivid depth sensor, and a human subjects study comparing performance on three variants of the peg-transfer task: unilateral, bilateral without handovers, and bilateral with handovers. The system combines 3D printing, depth sensing, and deep …
Purdue University（普渡大学）；Indiana University School of Medicine（印第安纳大学医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Short response time is critical for future military medical operations in austere settings or remote areas. Such effective patient care at the point of injury can greatly benefit from the integration of semi-autonomous robotic systems. To achieve autonomy, robots would require massive libraries of maneuvers. While this is possible in controlled settings, obtaining surgical data in austere settings can be difficult. Hence, in this paper, we present the Dexterous Surgical Skill (DESK) database for knowledge transfer between robots. The peg transfer task was selected as it is one of 6 main tasks …
Gaining spatial awareness of the Operating Room (OR) for surgical robotic systems is a key technology that can enable intelligent applications aiming at improved OR workflow. In this work, we present a method for semantic dense reconstruction of the OR scene using multiple RGB-D cameras attached and registered to the da Vinci Xi surgical system. We developed a novel SLAM approach for robot pose tracking in dynamic OR environments and dense reconstruction of the static OR table object. We validated our techniques in a mock OR by collecting data sequences with corresponding optical tracking traj…
Tissue manipulation is a frequently used fundamental subtask of any surgical procedures, and in some cases it may require the involvement of a surgeon's assistant. The complex dynamics of soft tissue as an unstructured environment is one of the main challenges in any attempt to automate the manipulation of it via a surgical robotic system. Two AI learning based model predictive control algorithms using vision strategies are proposed and studied: (1) reinforcement learning and (2) learning from demonstration. Comparison of the performance of these AI algorithms in a simulation setting indicated…
Gravity Compensation of the dVRK-Si Patient Side Manipulator based on Dynamic Model Identification
Department of Robotics Engineering, Worcester Polytechnic Institute（机器人工程系，沃斯通理工大学）；Laboratory for Computational Sensing and Robotics, JHU（计算感知与机器人实验室，约翰·霍普金斯大学）；Department of Computer Science, Vanderbilt University（计算机科学系，范德比尔特大学）；Department of Computer Science, Johns Hopkins University（计算机科学系，约翰·霍普金斯大学）
With introduction of new technologies in the operating room like the da Vinci Surgical System, training surgeons to use them effectively and efficiently is crucial in the delivery of better patient care. Coaching by an expert surgeon is effective in teaching relevant technical skills, but current methods to deliver effective coaching are limited and not scalable. We present a virtual reality simulation-based framework for automated virtual coaching in surgical education. We implement our framework within the da Vinci Skills Simulator. We provide three coaching modes ranging from a hands-on tea…
Diffusion Stabilizer Policy for Automated Surgical Robot Manipulations
UM-SJTU Joint Intuitute（UM-SJTU联合学院）；Shanghai Jiao Tong University（上海交通大学）；Department of Computer Science and Engineering（计算机科学与工程系）；The Chinese University of Hong Kong（香港中文大学）；Department of Automation（自动化系）
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
UC Berkeley（加州大学伯克利分校）；SRI International（SRI国际）；UC San Francisco Medical School（加州大学旧金山分校医学院）
Vascular shunt insertion is a fundamental surgical procedure used to temporarily restore blood flow to tissues. It is often performed in the field after major trauma. We formulate a problem of automated vascular shunt insertion and propose a pipeline to perform Automated Vascular Shunt Insertion (AVSI) using a da Vinci Research Kit. The pipeline uses a learned visual model to estimate the locus of the vessel rim, plans a grasp on the rim, and moves to grasp at that point. The first robot gripper then pulls the rim to stretch open the vessel with a dilation motion. The second robot gripper then…
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Shanghai Jiao Tong University（上海交通大学）
Surgical robots offer the exciting potential for remote telesurgery, but advances are needed to make this technology efficient and accurate to ensure patient safety. Achieving these goals is hindered by the deleterious effects of latency between the remote operator and the bedside robot. Predictive displays have found success in overcoming these effects by giving the operator immediate visual feedback. However, previously developed predictive displays can not be directly applied to telesurgery due to the unique challenges in tracking the 3D geometry of the surgical environment. In this paper, …
Overcoming Imperfect Kinematics in Surgical Robotics Through Sim-to-Real Visuomotor Learning
Hamlyn Centre for Robotic Surgery, Institute of Global Health Innovation, Imperial College London（伦敦帝国理工学院全球健康创新研究所哈姆林机器人手术中心）；Department of Surgery and Cancer, Imperial College London（伦敦帝国理工学院外科与癌症系）；Department of Mechanical Engineering, Imperial College London（伦敦帝国理工学院机械工程系）
Autonomy is the frontier of research in robotic surgery and its aim is to improve the quality of surgical procedures in the next future. One fundamental requirement for autonomy is advanced perception capability through vision sensors. In this paper, we propose a novel calibration technique for a surgical scenario with da Vinci robot. Calibration of the camera and the robot is necessary for precise positioning of the tools in order to emulate the high performance surgeons. Our calibration technique is tailored for RGB-D camera. Different tests performed on relevant use cases for surgery prove …
University of Bristol（布里斯托大学）；Bristol Robotics Lab（布里斯托机器人实验室）；Imperial College London（帝国理工学院）；Polytechnic University of Milan（米兰理工大学）；Shanghai Jiao Tong University（上海交通大学）；Worcester Polytechnic Institute（伍斯特理工学院）
Human-robot shared control, which integrates the advantages of both humans and robots, is an effective approach to facilitate efficient surgical operation. Learning from demonstration (LfD) techniques can be used to automate some of the surgical subtasks for the construction of the shared control framework. However, a sufficient amount of data is required for the robot to learn the manoeuvres. Using a surgical simulator to collect data is a less resource-demanding approach. With sim-to-real adaptation, the manoeuvres learned from a simulator can be transferred to a physical robot. To this end,…
University of California, Berkeley（加州大学伯克利分校）；UC San Francisco East Bay（加州大学旧金山分校东湾）；SRI International（斯坦福国际研究院）
Automation of surgical tasks using cable-driven robots is challenging due to backlash, hysteresis, and cable tension, and these issues are exacerbated as surgical instruments must often be changed during an operation. In this work, we propose a framework for automation of high-precision surgical tasks by learning sample efficient, accurate, closed-loop policies that operate directly on visual feedback instead of robot encoder estimates. This framework, which we call intermittent visual servoing (IVS), intermittently switches to a learned visual servo policy for high-precision segments of repet…
An Open-Source Robotics Research Platform for Autonomous Laparoscopic Surgery
Department of Translational Surgical Oncology, NCT/UCC Dresden, a partnership between DKFZ, MF UKD, TUD, HZDR Dresden, Germany（转化外科肿瘤学系，NCT/UCC德累斯顿，DKFZ、MF UKD、TUD、HZDR德累斯顿的联合体）；Cluster of Excellence-CeTI, TUD, Germany（卓越中心- CeTI，TUD，德国）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Germany（visceral、胸腔和血管外科系，医学学院和卡尔·格奥尔格·卡尔医院，TUD，德国）
Intuitive Surgical Inc.（直觉外科公司）；California Institute of Technology（加州理工学院）
This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual…
Laboratory for Computational Sensing and Robotics, Johns Hopkins University（约翰斯·霍普金斯大学计算感知与机器人实验室）
This paper presents a dynamic constraint formulation to provide protective virtual fixtures of 3D anatomical structures from polygon mesh representations. The proposed approach can anisotropically limit the tool motion of surgical robots without any assumption of the local anatomical shape close to the tool. Using a bounded search strategy and Principle Directed tree, the proposed system can run efficiently at 180 Hz for a mesh object containing 989,376 triangles and 493,460 vertices. The proposed algorithm has been validated in both simulation and skull cutting experiments. The skull cutting …
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
The restriction of feasible motions of a manipulator link constrained to move through an entry port is a common problem in minimum invasive surgery procedures. Additional spatial restrictions are required to ensure the safety of sensitive regions from unintentional damage. In this work, we design a target admittance model that is proved to enforce robot tool manipulation by a human through a remote center of motion and to guarantee that the tool will never enter or touch forbidden regions. The control scheme is proved passive under the exertion of a human force ensuring manipulation stability,…
Task automation of surgical robot has the potentials to improve surgical efficiency. Recent reinforcement learning (RL) based approaches provide scalable solutions to surgical automation, but typically require extensive data collection to solve a task if no prior knowledge is given. This issue is known as the exploration challenge, which can be alleviated by providing expert demonstrations to an RL agent. Yet, how to make effective use of demonstration data to improve exploration efficiency still remains an open challenge. In this work, we introduce Demonstration-guided EXploration (DEX), an e…
Phase-and-First-Arrival VLM Feedback for Sparse-Reward Reinforcement Learning in Surgical Manipulation
The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Great Bay University（大湾区大学）
Autonomy in robotic surgery is very challenging in unstructured environments, especially when interacting with deformable soft tissues. The main difficulty is to generate model-based control methods that account for deformation dynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes within the scene, however, model-based controllers integrated with dynamic properties, a more accurate and safe approach, has not been studied before. Considering the mechanic coupling between the robot and the environment, it is crucial to develop a register…
Control Modes of Teleoperated Surgical Robotic System's Tools in Ophthalmic Surgery
Department of Mechanical and Aerospace Engineering at the University of California, Los Angeles (UCLA)（加州大学洛杉矶分校机械与航空航天工程系）；Jules Stein Eye Institute at the University of California, Los Angeles (UCLA)（加州大学洛杉矶分校Jules Stein眼科研究所）
UC Berkeley（加州大学伯克利分校）；UCSF East Bay（加州大学旧金山分校东湾）；SRI International（斯坦福国际研究院）
Automation of surgical subtasks using cable-driven robotic surgical assistants (RSAs) such as Intuitive Surgical's da Vinci Research Kit (dVRK) is challenging due to imprecision in control from cable-related effects such as cable stretching and hysteresis. We propose a novel approach to efficiently calibrate such robots by placing a 3D printed fiducial coordinate frames on the arm and end-effector that is tracked using RGBD sensing. To measure the coupling and history-dependent effects between joints, we analyze data from sampled trajectories and consider 13 approaches to modeling. These model…
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
University of California, Los Angeles (UCLA)（加州大学洛杉矶分校（UCLA））
In this paper, we present a synergic learning algorithm to address the task of indirect manipulation of an unknown deformable tissue. Tissue manipulation is a common yet challenging task in various surgical interventions, which makes it a good candidate for robotic automation. We propose using a linear approximate Q-learning method in which human knowledge contributes to selecting useful yet simple features of tissue manipulation while the algorithm learns to take optimal actions and accomplish the task. The algorithm is implemented and evaluated on a simulation using the OpenCV and CHAI3D lib…
University College London（伦敦大学学院）；Welcome/EPSRC Centre for Interventional and Surgical Sciences（惠康/工程与物理科学研究委员会介入与外科科学中心）
In robotic-assisted partial nephrectomy, surgeons remove a part of a kidney often due to the presence of a mass. A drop-in ultrasound probe paired to a surgical robot is deployed to execute multiple swipes over the kidney surface to localise the mass and define the margins of resection. This sub-task is challenging and must be performed by a highly skilled surgeon. Automating this sub-task may reduce cognitive load for the surgeon and improve patient outcomes. The overall goal of this work is to autonomously move the ultrasound probe on the surface of the kidney taking advantage of the use of …
The Chinese University of Hong Kong（香港中文大学）；T Stone Robotics Institute, CUHK（香港中文大学T Stone机器人研究所）
Autonomous surgical execution relieves tedious routines and surgeon's fatigue. Recent learning-based methods, especially reinforcement learning (RL) based methods, achieve promising performance for dexterous manipulation, which usually requires the simulation to collect data efficiently and reduce the hardware cost. The existing learning-based simulation platforms for medical robots suffer from limited scenarios and simplified physical interactions, which degrades the real-world performance of learned policies. In this work, we designed SurRoL, an RL-centered simulation platform for surgical r…
