数据集在外科机器人领域正变得越来越重要，因为它们可用于识别和自动化任务。此外，这也使得使用公共数据集来比较不同算法和方法成为可能。本工作的目标是提供一个完整的数据集，包含外科医生为提高技能而执行的三项常见训练外科任务。为此，12名受试者遥操作达芬奇研究套件（da Vinci Research Kit）来完成这些任务。所获得的数据集包括达芬奇机器人（主端和从端）提供的所有运动学和动力学信息，以及来自摄像头的相关视频。所有信息都经过仔细的时间戳标记，并以可读的csv格式提供。还提供了一个与ROS集成的MATLAB接口，用于使用和复现这些数据。
英文摘要：
The use of datasets is getting more relevance in surgical robotics since they can be used to recognise and automate tasks. Also, this allows to use common datasets to compare different algorithms and methods. The objective of this work is to provide a complete dataset of three common training surgical tasks that surgeons perform to improve their skills. For this purpose, 12 subjects teleoperated the da Vinci Research Kit to perform these tasks. The obtained dataset includes all the kinematics and dynamics information provided by the da Vinci robot (both master and slave side) together with the associated video from the camera. All the information has been carefully timestamped and provided in a readable csv format. A MATLAB interface integrated with ROS for using and replicating the data is also provided.
Imperial College London（伦敦帝国理工学院）；University of Leeds（利兹大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Worcester Polytechnic Institute（伍斯特理工学院）；Intuitive Surgical Inc（直觉外科公司）
Robotic-assisted surgery is now well-established in clinical practice and has become the gold standard clinical treatment option for several clinical indications. The field of robotic-assisted surgery is expected to grow substantially in the next decade with a range of new robotic devices emerging to address unmet clinical needs across different specialities. A vibrant surgical robotics research community is pivotal for conceptualizing such new systems as well as for developing and training the engineers and scientists to translate them into practice. The da Vinci Research Kit (dVRK), an acade…
Overcoming Imperfect Kinematics in Surgical Robotics Through Sim-to-Real Visuomotor Learning
Hamlyn Centre for Robotic Surgery, Institute of Global Health Innovation, Imperial College London（伦敦帝国理工学院全球健康创新研究所哈姆林机器人手术中心）；Department of Surgery and Cancer, Imperial College London（伦敦帝国理工学院外科与癌症系）；Department of Mechanical Engineering, Imperial College London（伦敦帝国理工学院机械工程系）
University of Bristol（布里斯托大学）；Bristol Robotics Lab（布里斯托机器人实验室）；Imperial College London（帝国理工学院）；Polytechnic University of Milan（米兰理工大学）；Shanghai Jiao Tong University（上海交通大学）；Worcester Polytechnic Institute（伍斯特理工学院）
Human-robot shared control, which integrates the advantages of both humans and robots, is an effective approach to facilitate efficient surgical operation. Learning from demonstration (LfD) techniques can be used to automate some of the surgical subtasks for the construction of the shared control framework. However, a sufficient amount of data is required for the robot to learn the manoeuvres. Using a surgical simulator to collect data is a less resource-demanding approach. With sim-to-real adaptation, the manoeuvres learned from a simulator can be transferred to a physical robot. To this end,…
University of California, Berkeley（加利福尼亚大学伯克利分校）；UC San Francisco East Bay（加州大学旧金山分校东湾校区）；SRI International（斯坦福国际研究院）
Recent advances in depth-sensing have significantly increased accuracy, resolution, and frame rate, as shown in the 1920x1200 resolution and 13 frames per second Zivid RGBD camera. In this study, we explore the potential of depth sensing for efficient and reliable automation of surgical subtasks. We consider a monochrome (all red) version of the peg transfer task from the Fundamentals of Laparoscopic Surgery training suite implemented with the da Vinci Research Kit (dVRK). We use calibration techniques that allow the imprecise, cable-driven da Vinci to reduce error from 4-5 mm to 1-2 mm in the…
On-the-fly hand-eye calibration for the da Vinci surgical robot
Department of Mechanical Engineering, Imperial College London（帝国理工学院机械工程系）；Mechatronics in Medicine Laboratory（医学机电实验室）；Hamlyn Centre for Robotics Surgery（机器人外科哈姆林中心）
The da Vinci Research Kit (dVRK) is a teleoperated surgical robotic system. For dynamic simulations and model-based control, the dynamic model of the dVRK is required. We present an open-source dynamic model identification package for the dVRK, capable of modeling the parallelograms, springs, counterweight, and tendon couplings, which are inherent to the dVRK. A convex optimization-based method is used to identify the dynamic parameters of the dVRK subject to physical consistency. Experimental results show the effectiveness of the modeling and the robustness of the package. Although this softw…
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
Intuitive Surgical Inc.（直觉外科公司）；California Institute of Technology（加州理工学院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state est…
SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics
Department of Robotics Engineering, Worcester Polytechnic Institute（机器人工程系，沃斯特理工学院）；Laboratory for Computational Sensing and Robotics（计算感知与机器人实验室）；Department of Computer Science, Johns Hopkins University（计算机科学系，约翰霍普金斯大学）；Robotics and Control Laboratory, the University of British Columbia（机器人与控制实验室，不列颠哥伦比亚大学）；Department of Electronics, Information and Bioengineering, Politecnico di Milano（电子、信息与生物工程系，米兰理工学院）
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Kyung Hee University（庆熙大学）
Innovations from surgical robotic research rarely translates to live surgery due to the significant difference between the lab and a live environment. Live environments require considerations that are often overlooked during early stages of research such as surgical staff, surgical procedure, and the challenges of working with live tissue. One such example is the da Vinci Research Kit (dVRK) which is used by over 40 robotics research groups and represents an open-sourced version of the da Vinci Surgical System. Despite dVRK being available for nearly a decade and the ideal candidate for transl…
Learning Autonomous Surgical Irrigation and Suction with the da Vinci Research Kit Using Reinforcement Learning
Department of Electrical and Computer Engineering, University of Alberta（电气与计算机工程系，阿尔伯塔大学）；Department of Biomedical Engineering, University of Alberta（生物医学工程系，阿尔伯塔大学）
University of California, Berkeley（加州大学伯克利分校）；SRI International（SRI国际）
Peg transfer is a well-known surgical training task in the Fundamentals of Laparoscopic Surgery (FLS). While human sur-geons teleoperate robots such as the da Vinci to perform this task with high speed and accuracy, it is challenging to automate. This paper presents a novel system and control method using a da Vinci Research Kit (dVRK) surgical robot and a Zivid depth sensor, and a human subjects study comparing performance on three variants of the peg-transfer task: unilateral, bilateral without handovers, and bilateral with handovers. The system combines 3D printing, depth sensing, and deep …
College of Medicine, University of Illinois Chicago（伊利诺伊大学芝加哥分校医学院）；School of Medicine and Surgery, University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学医学与外科学院）；Department of Medicine and Surgery, University of Milan（米兰大学医学与外科学院）；Lugano Regional Hospital, Ente Ospedaliero Cantonale (EOC)（卢加诺地区医院（卢加诺州立医院协会））；College of Engineering, University of Illinois Chicago（伊利诺伊大学芝加哥分校工学院）
Measurement of environment interaction forces during robotic minimally-invasive surgery would enable haptic feedback to the surgeon, thereby solving one long-standing limitation. Estimating this force from existing sensor data avoids the challenge of retrofitting systems with force sensors, but is difficult due to mechanical effects such as friction and compliance in the robot mechanism. We have previously shown that neural networks can be trained to estimate the internal robot joint torques, thereby enabling estimation of external forces. In this work, we extend the method to estimate externa…
College of Medicine, University of Illinois Chicago（伊利诺伊大学芝加哥分校医学院）；School of Medicine and Surgery, University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学医学与外科学院）；Department of Medicine and Surgery, University of Milan（米兰大学医学与外科学院）；Lugano Regional Hospital, Ente Ospedaliero Cantonale (EOC)（卢加诺地区医院（坎顿医院基金会））；College of Engineering, University of Illinois Chicago（伊利诺伊大学芝加哥分校工程学院）
Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics
Open-H-Embodiment Consortium；University of California, Berkeley（加州大学伯克利分校）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Southern California（南加州大学）；University of Cambridge（剑桥大学）；University of Tokyo（东京大学）；University of Tokyo, Graduate School of Information Science and Technology（东京大学信息科学与技术研究生院）；University of Tokyo, Institute of Industrial Science（东京大学工业科学研究所）
Gravity Compensation of the dVRK-Si Patient Side Manipulator based on Dynamic Model Identification
Department of Robotics Engineering, Worcester Polytechnic Institute（机器人工程系，沃斯通理工大学）；Laboratory for Computational Sensing and Robotics, JHU（计算感知与机器人实验室，约翰·霍普金斯大学）；Department of Computer Science, Vanderbilt University（计算机科学系，范德比尔特大学）；Department of Computer Science, Johns Hopkins University（计算机科学系，约翰·霍普金斯大学）
With introduction of new technologies in the operating room like the da Vinci Surgical System, training surgeons to use them effectively and efficiently is crucial in the delivery of better patient care. Coaching by an expert surgeon is effective in teaching relevant technical skills, but current methods to deliver effective coaching are limited and not scalable. We present a virtual reality simulation-based framework for automated virtual coaching in surgical education. We implement our framework within the da Vinci Skills Simulator. We provide three coaching modes ranging from a hands-on tea…
Robotic Surgical Assistants (RSAs) are commonly used to perform minimally invasive surgeries by expert surgeons. However, long procedures filled with tedious and repetitive tasks such as suturing can lead to surgeon fatigue, motivating the automation of suturing. As visual tracking of a thin reflective needle is extremely challenging, prior work has modified the needle with nonreflective contrasting paint. As a step towards automation of a suturing subtask without modifying the needle, we propose HOUSTON: Handoff of Unmodified, Surgical, Tool-Obstructed Needles, a problem and algorithm that us…
An Open-Source Robotics Research Platform for Autonomous Laparoscopic Surgery
Department of Translational Surgical Oncology, NCT/UCC Dresden, a partnership between DKFZ, MF UKD, TUD, HZDR Dresden, Germany（转化外科肿瘤学系，NCT/UCC德累斯顿，DKFZ、MF UKD、TUD、HZDR德累斯顿的联合体）；Cluster of Excellence-CeTI, TUD, Germany（卓越中心- CeTI，TUD，德国）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Germany（visceral、胸腔和血管外科系，医学学院和卡尔·格奥尔格·卡尔医院，TUD，德国）
The Chinese University of Hong Kong（香港中文大学）；Imperial College London（伦敦帝国学院）
Surgical knot tying is one of the most fundamental and important procedures in surgery, and a high-quality knot can significantly benefit the postoperative recovery of the patient. However, a longtime operation may easily cause fatigue to surgeons, especially during the tedious wound closure task. In this paper, we present a vision-based method to automate the suture thread grasping, which is a sub-task in surgical knot tying and an intermediate step between the stitching and looping manipulations. To achieve this goal, the acquisition of a suture's three-dimensional (3D) information is critic…
Task automation of surgical robot has the potentials to improve surgical efficiency. Recent reinforcement learning (RL) based approaches provide scalable solutions to surgical automation, but typically require extensive data collection to solve a task if no prior knowledge is given. This issue is known as the exploration challenge, which can be alleviated by providing expert demonstrations to an RL agent. Yet, how to make effective use of demonstration data to improve exploration efficiency still remains an open challenge. In this work, we introduce Demonstration-guided EXploration (DEX), an e…
As surgical robots become more common, automating away some of the burden of complex direct human operation becomes ever more feasible. Model-free reinforcement learning (RL) is a promising direction toward generalizable automated surgical performance, but progress has been slowed by the lack of efficient and realistic learning environments. In this paper, we describe adding reinforcement learning support to the da Vinci Skill Simulator, a training simulation used around the world to allow surgeons to learn and rehearse technical skills. We successfully teach an RL-based agent to perform sub-t…
Haptic Intelligence Department（人机交互智能部门）；Optics and Sensing Laboratory, Max Planck Institute for Intelligent Systems（光学与感知实验室，智能系统马克斯·普朗克研究所）；Department of General, Visceral and Transplant Surgery, University Hospital Tübingen（普通外科、 visceral外科和移植外科部门，图宾根大学医院）；Laboratory for Computational Sensing and Robotics (LCSR) Johns Hopkins University（计算感知与机器人实验室（LCSR）约翰霍普金斯大学）
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery. With recent advancement in robot-assisted minimally invasive surgery, rich information including surgical videos and robotic kinematics can be recorded, which provide complementary knowledge for understanding surgical gestures. However, existing methods either solely adopt uni-modal data or directly concatenate multi-modal representations, which can not sufficiently exploit the informative correlations inherent in visual and kinematics data to boost gesture recogniti…
