Autonomy in robotic surgery is very challenging in unstructured environments, especially when interacting with deformable soft tissues. The main difficulty is to generate model-based control methods that account for deformation dynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes within the scene, however, model-based controllers integrated with dynamic properties, a more accurate and safe approach, has not been studied before. Considering the mechanic coupling between the robot and the environment, it is crucial to develop a registered, simulated dynamical model. In this work, we propose an online, continuous, real-to-sim registration method to bridge 3D visual perception with position-based dynamics (PBD) modeling of tissues. The PBD method is employed to simulate soft tissue dynamics as well as rigid tool interactions for model-based control. Meanwhile, a vision-based strategy is used to generate 3D reconstructed point cloud surfaces based on real-world manipulation, so as to register and update the simulation. To verify this real-to-sim approach, tissue experiments have been conducted on the da Vinci Research Kit. Our real-to-sim approach successfully reduces registration error online, which is especially important for safety during autonomous control. Moreover, it achieves higher accuracy in occluded areas than fusion-based reconstruction.
Autonomous surgery involves having surgical tasks performed by a robot operating under its own will, with partial or no human involvement. There are several important advantages of automation in surgery, which include increasing precision of care due to sub-millimeter robot control, real-time utilization of biosignals for interventional care, improvements to surgical efficiency and execution, and computer-aided guidance under various medical imaging and sensing modalities. While these methods may displace some tasks of surgical teams and individual surgeons, they also present new capabilities …
Robotic surgery has increased the domain of surgeries possible. Several examples of partial surgical automation have been seen in the past decade. We break down the path of automation tasks into features required and provide a checklist that can help reach higher levels of surgical automation. Finally, we discuss the current challenges and advances required to make this happen.
SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy
Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Department of Thoracic Surgery, Peking University People’s Hospital（北京大学人民医院胸外科）；Thoracic Oncology Institute, Peking University People’s Hospital（北京大学人民医院胸部肿瘤研究所）；Research Unit of Intelligence Diagnosis and Treatment in Early Non-small Cell Lung Cancer, Chinese Academy of Medical Sciences（中国医学科学院早期非小细胞肺癌智能诊断与治疗研究组）；Institute of Advanced Clinical Medicine, Peking University（北京大学先进临床医学院）；Beijing Key Laboratory of Innovative Application of Big Data in Lung Cancer, Peking University People’s Hospital（北京大学人民医院肺癌大数据创新应用北京市重点实验室）
Simultaneous Estimation of Shape and Force along Highly Deformable Surgical Manipulators Using Sparse FBG Measurement
T Stone Robotics Institute（T Stone机器人研究所）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong Center for Logistics Robotics（香港物流机器人中心）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）
University of California, Los Angeles (UCLA)（加州大学洛杉矶分校（UCLA））
In this paper, we present a synergic learning algorithm to address the task of indirect manipulation of an unknown deformable tissue. Tissue manipulation is a common yet challenging task in various surgical interventions, which makes it a good candidate for robotic automation. We propose using a linear approximate Q-learning method in which human knowledge contributes to selecting useful yet simple features of tissue manipulation while the algorithm learns to take optimal actions and accomplish the task. The algorithm is implemented and evaluated on a simulation using the OpenCV and CHAI3D lib…
Autonomy in robot-assisted surgery is essential to reduce surgeons' cognitive load and eventually improve the overall surgical outcome. A key requirement for autonomy in a safety-critical scenario as surgery lies in the generation of interpretable plans that rely on expert knowledge. Moreover, the Autonomous Robotic Surgical System (ARSS) must be able to reason on the dynamic and unpredictable anatomical environment, and quickly adapt the surgical plan in case of unexpected situations. In this paper, we present a modular Framework for Robot-Assisted Surgery (FRAS) in deformable anatomical envi…
Autonomy is the frontier of research in robotic surgery and its aim is to improve the quality of surgical procedures in the next future. One fundamental requirement for autonomy is advanced perception capability through vision sensors. In this paper, we propose a novel calibration technique for a surgical scenario with da Vinci robot. Calibration of the camera and the robot is necessary for precise positioning of the tools in order to emulate the high performance surgeons. Our calibration technique is tailored for RGB-D camera. Different tests performed on relevant use cases for surgery prove …
AssistDLO: Assistive Teleoperation for Deformable Linear Object Manipulation
TU Darmstadt, Institute for Intelligent Autonomous Systems, Computer Science Department（德累斯顿技术大学智能自主系统研究所，计算机科学系）；Honda Research Institute Europe GmbH（本田欧洲研究院）；German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning（德国人工智能研究中心（DFKI），研究部门：机器人学习的系统AI；黑森AI；德国机器人研究所（RIG）；认知科学中心）；Hessian.AI；Robotics Institute Germany (RIG)；Centre for Cognitive Science
Johns Hopkins University（约翰斯·霍普金斯大学）；Children’s National Hospital（国立儿童医院）
Autonomous robotic surgery has the potential to provide efficacy, safety, and consistency independent of individual surgeons skill and experience. Autonomous soft-tissue surgery in unstructured and deformable environments is especially challenging as it necessitates intricate imaging, tissue tracking and surgical planning techniques, as well as a precise execution via highly adaptable control strategies. In the laparoscopic setting, soft-tissue surgery is even more challenging due to the need for high maneuverability and repeatability under motion and vision constraints. We demonstrate the fir…
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
Institute for Intelligent Systems Research and Innovation, Deakin University（迪肯大学智能系统研究与创新研究所）；Imperial College London（伦敦帝国理工学院）
Surgeons normally need surgical scissors and tissue grippers to cut through a deformable surgical tissue. The cutting accuracy depends on the skills to manipulate these two tools. Such skills are part of basic surgical skills training as in the Fundamentals of Laparoscopic Surgery. The gripper is used to pinch a point on the surgical sheet and pull the tissue to a certain direction to maintain the tension while the scissors cut through a trajectory. As the surgical materials are deformable, it requires a comprehensive tensioning policy to yield appropriate tensioning direction at each step of …
University of Utah（犹他大学）；NVIDIA Corporation（英伟达公司）
If robots could reliably manipulate the shape of 3D deformable objects, they could find applications in fields ranging from home care to warehouse fulfillment to surgical assistance. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural ne…
University of Bologna（博洛尼亚大学）；Menoufia University（曼努菲亚大学）
Deformable Linear Objects (DLOs) such as ropes, cables, and surgical sutures have a wide variety of uses in automotive engineering, surgery, and electromechanical industries. Therefore, modeling of DLOs as well as a computationally efficient way to predict the DLO behavior are of great importance, in particular to enable robotic manipulation of DLOs. The main motivation of this work is to enable efficient prediction of the DLO behavior during robotic manipulation. In this paper, the DLO is modeled by a multivariate dynamic spline, while a symplectic integration method is used to solve the mode…
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics
Department of Robotics Engineering, Worcester Polytechnic Institute（机器人工程系，沃斯特理工学院）；Laboratory for Computational Sensing and Robotics（计算感知与机器人实验室）；Department of Computer Science, Johns Hopkins University（计算机科学系，约翰霍普金斯大学）；Robotics and Control Laboratory, the University of British Columbia（机器人与控制实验室，不列颠哥伦比亚大学）；Department of Electronics, Information and Bioengineering, Politecnico di Milano（电子、信息与生物工程系，米兰理工学院）
Purdue University（普渡大学）；Indiana University School of Medicine（印第安纳大学医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Short response time is critical for future military medical operations in austere settings or remote areas. Such effective patient care at the point of injury can greatly benefit from the integration of semi-autonomous robotic systems. To achieve autonomy, robots would require massive libraries of maneuvers. While this is possible in controlled settings, obtaining surgical data in austere settings can be difficult. Hence, in this paper, we present the Dexterous Surgical Skill (DESK) database for knowledge transfer between robots. The peg transfer task was selected as it is one of 6 main tasks …
Intuitive Surgical Inc.（直觉外科公司）；California Institute of Technology（加州理工学院）
This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual…
ArthroCut: Autonomous Policy Learning for Robotic Bone Resection in Knee Arthroplasty
School of Biomedical Engineering, Tsinghua University（清华大学生物医学工程学院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Longwood Valley MedTech
Contact-Stable Deformable Tissue Simulation Using Implicit Integration and Live-Pose Grasp Constraints for Laparoscopic Surgery Robot Policy Evaluation
Seoul National University Hospital（首尔大学医院）；Rosota Inc.（Rosota公司）；Seoul National University College of Medicine（首尔大学医学院）；Seoul National University（首尔大学）；Institute of Convergence Medicine with Innovative Technology, Seoul National University Hospital（首尔大学医院创新技术融合医学研究所）；Eulji University College of Medicine（乙支大学医学院）；Chungang University（中央大学）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Heidelberg University Hospital（海德堡大学医院）；Friedrich-Alexander-University Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Technical University Dresden（德累斯顿工业大学）
Recent advances in reinforcement learning (RL) have increased the promise of introducing cognitive assistance and automation to robot-assisted laparoscopic surgery (RALS). However, progress in algorithms and methods depends on the availability of standardized learning environments that represent skills relevant to RALS. We present LapGym, a framework for building RL environments for RALS that models the challenges posed by surgical tasks, and sofa_env, a diverse suite of 12 environments. Motivated by surgical training, these environments are organized into 4 tracks: Spatial Reasoning, Deformab…
Registering Neural 4D Gaussians for Endoscopic Surgery
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Shenzhen Research Institute, CUHK（香港中文大学深圳研究院）；KTH, Kungliga Tekniska högskolan Royal Institute of Technology（瑞典皇家理工学院）；Hong Kong Applied Science and Technology Research Institute Company Limited（香港应用科技研究院有限公司）
Shaft-integrated Force Sensing with Transformer-based Dynamics Compensation for Telesurgery
Department of Electrical, Computer, and Systems Engineering, Case Western Reserve University（电气、计算机与系统工程系，凯斯西储大学）；Department of Mechanical and Aerospace Engineering, Case Western Reserve University（机械与航空航天工程系，凯斯西储大学）；Resense GmbH；Semantic Information Systems Group, Osnabrück University（语义信息系统组，奥斯纳布吕克大学）；Justus Liebig University（吉森大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
Haptic Intelligence Department（人机交互智能部门）；Optics and Sensing Laboratory, Max Planck Institute for Intelligent Systems（光学与感知实验室，智能系统马克斯·普朗克研究所）；Department of General, Visceral and Transplant Surgery, University Hospital Tübingen（普通外科、 visceral外科和移植外科部门，图宾根大学医院）；Laboratory for Computational Sensing and Robotics (LCSR) Johns Hopkins University（计算感知与机器人实验室（LCSR）约翰霍普金斯大学）
Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing
Technische Universität München（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute（威尔默眼科研究所）；Johns Hopkins Hospital（约翰斯·霍普金斯医院）；Technical University of Munich（慕尼黑工业大学）；Whiting School of Engineering（怀廷工程学院）
University of Verona（维罗纳大学）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
The current research focus in Robot-Assisted Minimally Invasive Surgery (RAMIS) is directed towards increasing the level of robot autonomy, to place surgeons in a supervisory position. Although Learning from Demonstrations (LfD) approaches are among the preferred ways for an autonomous surgical system to learn expert gestures, they require a high number of demonstrations and show poor generalization to the variable conditions of the surgical environment. In this work, we propose an LfD methodology based on Generative Adversarial Imitation Learning (GAIL) that is built on a Deep Reinforcement L…
