Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and reduced feature engineering efforts to fully perceive a surgical scene. The framework was tested on three publicly available datasets, which use the da Vinci Surgical System, for comprehensive analysis. Experimental results show that our framework achieves state-of-the-art tracking performance in a surgical environment by utilizing deep learning for feature extraction.
Imperial College London（帝国理工学院）；German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）；Charing Cross Hospital（查令十字医院）
The aim of this paper is to introduce a robotic platform for autonomous iUS tissue scanning to optimise intraoperative diagnosis and improve surgical resection during robot-assisted operations. To guide anatomy specific robotic scanning and generate a representation of the robot task space, fast and accurate techniques for the recovery of 3D morphological structures of the surgical cavity are developed. The prototypic DLR MIRO surgical robotic arm is used to control the applied force and the in-plane motion of the US transducer. A key application of the proposed platform is the scanning of bra…
Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University College London（伦敦大学学院）；The Hong Kong University of Science and Technology（香港科技大学）
Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）
Providing force feedback as relevant information in current Robot-Assisted Minimally Invasive Surgery systems constitutes a technological challenge due to the constraints imposed by the surgical environment. In this context, Sensorless Force Estimation techniques represent a potential solution, enabling to sense the interaction forces between the surgical instruments and soft-tissues. Specifically, if visual feedback is available for observing soft-tissues' deformation, this feedback can be used to estimate the forces applied to these tissues. To this end, a force estimation model, based on Co…
The control and task automation of robotic surgical system is very challenging, especially in soft tissue manipulation, due to the unpredictable deformations. Thus, an accurate simulator of soft tissues with the ability of interacting with robot manipulators is necessary. In this work, we propose a novel 2D simulation framework for tool-tissue interaction. This framework continuously tracks the motion of manipulator and simulates the tissue deformation in presence of collision detection. The deformation energy can be computed for the control and planning task.
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
Tissue manipulation is a frequently used fundamental subtask of any surgical procedures, and in some cases it may require the involvement of a surgeon's assistant. The complex dynamics of soft tissue as an unstructured environment is one of the main challenges in any attempt to automate the manipulation of it via a surgical robotic system. Two AI learning based model predictive control algorithms using vision strategies are proposed and studied: (1) reinforcement learning and (2) learning from demonstration. Comparison of the performance of these AI algorithms in a simulation setting indicated…
University of California, Los Angeles (UCLA)（加州大学洛杉矶分校（UCLA））
In this paper, we present a synergic learning algorithm to address the task of indirect manipulation of an unknown deformable tissue. Tissue manipulation is a common yet challenging task in various surgical interventions, which makes it a good candidate for robotic automation. We propose using a linear approximate Q-learning method in which human knowledge contributes to selecting useful yet simple features of tissue manipulation while the algorithm learns to take optimal actions and accomplish the task. The algorithm is implemented and evaluated on a simulation using the OpenCV and CHAI3D lib…
Purpose Surgical simulations play an increasingly important role in surgeon education and developing algorithms that enable robots to perform surgical subtasks. To model anatomy, Finite Element Method (FEM) simulations have been held as the gold standard for calculating accurate soft-tissue deformation. Unfortunately, their accuracy is highly dependent on the simulation parameters, which can be difficult to obtain. Methods In this work, we investigate how live data acquired during any robotic endoscopic surgical procedure may be used to correct for inaccurate FEM simulation results. Since FEMs…
Bayesian Retraction Optimization for Tissue Attachment Mapping in Surgical Dissection
The Robotics Center and the Kahlert School of Computing at the University of Utah（美国犹他大学机器人中心和卡尔特计算学院）；School of Computational Science and Engineering, Georgia Institute of Technology（美国佐治亚理工学院计算科学与工程学院）；Department of Electrical and Computer Engineering and Department of Computer Science, Vanderbilt University（美国范德堡大学电气与计算机工程系和计算机科学系）
Autonomy in robot-assisted surgery is essential to reduce surgeons' cognitive load and eventually improve the overall surgical outcome. A key requirement for autonomy in a safety-critical scenario as surgery lies in the generation of interpretable plans that rely on expert knowledge. Moreover, the Autonomous Robotic Surgical System (ARSS) must be able to reason on the dynamic and unpredictable anatomical environment, and quickly adapt the surgical plan in case of unexpected situations. In this paper, we present a modular Framework for Robot-Assisted Surgery (FRAS) in deformable anatomical envi…
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland, College Park（马里兰大学帕克分校）；Weinberg Medical Physics, Inc.（温伯格医学物理公司）
Real-time visual localization of needles is necessary for various surgical applications, including surgical automation and visual feedback. In this study we investigate localization and autonomous robotic control of needles in the context of our magneto-suturing system. Our system holds the potential for surgical manipulation with the benefit of minimal invasiveness and reduced patient side effects. However, the non-linear magnetic fields produce unintuitive forces and demand delicate position-based control that exceeds the capabilities of direct human manipulation. This makes automatic needle…
Autonomy in robotic surgery is very challenging in unstructured environments, especially when interacting with deformable soft tissues. The main difficulty is to generate model-based control methods that account for deformation dynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes within the scene, however, model-based controllers integrated with dynamic properties, a more accurate and safe approach, has not been studied before. Considering the mechanic coupling between the robot and the environment, it is crucial to develop a register…
MV-dVRK: A Multi-Viewpoint Benchmark for Spatial Surgical Perception
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zürich（苏黎世联邦理工学院）；Erbe Group（爱尔博集团）；Tübingen University Hospital（蒂宾根大学医院）；Johns Hopkins University（约翰霍普金斯大学）
Johns Hopkins University（约翰斯·霍普金斯大学）；Children’s National Hospital（国立儿童医院）
Autonomous robotic surgery has the potential to provide efficacy, safety, and consistency independent of individual surgeons skill and experience. Autonomous soft-tissue surgery in unstructured and deformable environments is especially challenging as it necessitates intricate imaging, tissue tracking and surgical planning techniques, as well as a precise execution via highly adaptable control strategies. In the laparoscopic setting, soft-tissue surgery is even more challenging due to the need for high maneuverability and repeatability under motion and vision constraints. We demonstrate the fir…
Institute for Intelligent Systems Research and Innovation, Deakin University（迪肯大学智能系统研究与创新研究所）；Imperial College London（伦敦帝国理工学院）
Surgeons normally need surgical scissors and tissue grippers to cut through a deformable surgical tissue. The cutting accuracy depends on the skills to manipulate these two tools. Such skills are part of basic surgical skills training as in the Fundamentals of Laparoscopic Surgery. The gripper is used to pinch a point on the surgical sheet and pull the tissue to a certain direction to maintain the tension while the scissors cut through a trajectory. As the surgical materials are deformable, it requires a comprehensive tensioning policy to yield appropriate tensioning direction at each step of …
Recent developments in surgical robotics have led to new advancements in the automation of surgical sub-tasks such as suturing, soft tissue manipulation, tissue tensioning and cutting. However, integration of dynamics to optimize these control policies for the variety of scenes encountered in surgery remains unsolved. Towards this effort, we investigate the integration of differentiable fluid dynamics to optimizing a suction tool's trajectory to clear the surgical field from blood as fast as possible. The fully differentiable fluid dynamics is integrated with a novel suction model for effectiv…
Capstan-driven Continuum Surgical Robot: Design, Modeling, and Perception
The Chinese University of Hong Kong（香港中文大学）；School of Mechanical and Electronic Engineering, Wuhan University of Technology（武汉理工大学机电工程学院）；Huazhong University of Science and Technology（华中科技大学）；Shun Hing Institute of Advanced Engineering（信兴高等工程研究院）；Multi-Scale Medical Robotics Center（多尺度医疗机器人中心）
TumorMap: A Laser-based Surgical Platform for 3D Tumor Mapping and Fully-Automated Tumor Resection
Thomas Lord Department of Mechanical Engineering and Materials Science, Duke University（杜克大学机械工程与材料科学系）；Department of Robotics, University of Michigan, Ann Arbor（密歇根大学机器人学系）；Department of Orthopaedic Surgery, School of Medicine, Duke University（杜克大学骨科手术系）；Department of Pathology, School of Medicine, Duke University（杜克大学病理学系）；Department of Ophthalmology and Visual Sciences, University of Michigan Medical School, Ann Arbor（密歇根大学医学学院眼科与视觉科学系）；Department of Neurosurgery, School of Medicine, Duke University（杜克大学神经外科系）
Design of an innovative robotic surgical instrument for circular stapling
University of Medicine and Pharmacy “Iuliu Hațieganu”（“尤利乌·哈蒂埃加努”医药大学）；École Centrale Nantes（南特中央理工学校）；Nantes Université（南特大学）；Technical Sciences Academy of Romania（罗马尼亚技术科学院）
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）
Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issues caused by limited demonstrations. In this paper, we propose a novel Imitation Learning framework for Laparoscope Control (ILLC) with reinforcement learning (RL), which can efficiently learn the con…
National University of Singapore（新加坡国立大学）；Imperial College London（伦敦帝国理工学院）；Singapore General Hospital（新加坡中央医院）；The Chinese University of Hong Kong（香港中文大学）；Shun Hing Institute of Advanced Engineering（信兴高等工程研究所）
Purpose: Surgery scene understanding with tool-tissue interaction recognition and automatic report generation can play an important role in intra-operative guidance, decision-making and postoperative analysis in robotic surgery. However, domain shifts between different surgeries with inter and intra-patient variation and novel instruments' appearance degrade the performance of model prediction. Moreover, it requires output from multiple models, which can be computationally expensive and affect real-time performance. Methodology: A multi-task learning (MTL) model is proposed for surgical report…
Low-Contact Grasping of Soft Tissue with Complex Geometry using a Vortex Gripper
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本国立先进工业科学技术研究院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
