Virtual Reality for Synergistic Surgical Training and Data Generation
Johns Hopkins University（约翰斯·霍普金斯大学）；Hospital for Sick Children（病童医院）
Adnan Munawar, Zhaoshuo Li, Punit Kunjam, Nimesh Nagururu, Andy S. Ding, Peter Kazanzides, Thomas Looi, Francis X. Creighton, Russell H. Taylor, Mathias Unberath
Surgical simulators not only allow planning and training of complex procedures, but also offer the ability to generate structured data for algorithm development, which may be applied in image-guided computer assisted interventions. While there have been efforts on either developing training platforms for surgeons or data generation engines, these two features, to our knowledge, have not been offered together. We present our developments of a cost-effective and synergistic framework, named Asynchronous Multibody Framework Plus (AMBF+), which generates data for downstream algorithm development simultaneously with users practicing their surgical skills. AMBF+ offers stereoscopic display on a virtual reality (VR) device and haptic feedback for immersive surgical simulation. It can also generate diverse data such as object poses and segmentation maps. AMBF+ is designed with a flexible plugin setup which allows for unobtrusive extension for simulation of different surgical procedures. We show one use case of AMBF+ as a virtual drilling simulator for lateral skull-base surgery, where users can actively modify the patient anatomy using a virtual surgical drill. We further demonstrate how the data generated can be used for validating and training downstream computer vision algorithms
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Shanghai Jiao Tong University（上海交通大学）
Surgical robots offer the exciting potential for remote telesurgery, but advances are needed to make this technology efficient and accurate to ensure patient safety. Achieving these goals is hindered by the deleterious effects of latency between the remote operator and the bedside robot. Predictive displays have found success in overcoming these effects by giving the operator immediate visual feedback. However, previously developed predictive displays can not be directly applied to telesurgery due to the unique challenges in tracking the 3D geometry of the surgical environment. In this paper, …
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）；Tsinghua University（清华大学）；Beijing Tsinghua Changgung Hospital（北京清华长庚医院）；Technical University of Munich（慕尼黑工业大学）
The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the task performance. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-loca…
The Chinese University of Hong Kong（香港中文大学）；The Johns Hopkins University（约翰斯·霍普金斯大学）
Robot-assisted surgery has become progressively more and more popular due to its clinical advantages. In the meanwhile, the artificial intelligence and augmented reality in robotic surgery are developing rapidly and receive lots of attention. However, current methods have not discussed the coherent integration of AI and AR in robotic surgery. In this paper, we develop a novel system by seamlessly merging artificial intelligence module and augmented reality visualization to automatically generate the surgical guidance for robotic surgery education. Specifically, we first leverage reinforcement …
Mixed-Reality-Guided Teleoperation of a Collaborative Robot for Surgical Procedures
Technical University of Cluj-Napoca（克卢日-纳波卡技术大学）；Technical Sciences Academy of Romania（罗马尼亚技术科学院）；”Iuliu Hatieganu” University of Medicine and Pharmacy（“尤利乌·哈蒂埃加努”医药大学）；École Centrale Nantes（南特中央理工学校）；Nantes Université（南特大学）；CNRS（法国国家科学研究中心）
本文提出一种结合 Kuka IIWA LBR 协作机器人与 Hololens 2 的混合现实遥操作系统，允许外科医生通过手势直观控制机器人定位，有效提升手术工作流程并缩短学习曲线。
An innovative mixed reality approach for Robotics Surgery
Technical University of Cluj-Napoca（克卢日-纳波卡技术大学）；Nantes Université（南特大学）；École Centrale Nantes（南特中央理工学院）；CNRS（法国国家科学研究中心）；LS2N（南特数字科学实验室）；Technical Sciences Academy of Romania（罗马尼亚技术科学院）
Charles Sturt University（查尔斯特大学）；Worcester State University（伍斯特州立大学）；University of Kurdistan Hewler（库尔德斯坦赫勒大学）；University of New South Wales（新南威尔士大学）；Greater Western Sydney Area Health Services（大西悉尼地区卫生服务）；Central Coast Area Health（中央海岸地区卫生局）
Background and aim: Most of the Mixed Reality models used in the surgical telepresence are suffering from discrepancies in the boundary area and spatial-temporal inconsistency due to the illumination variation in the video frames. The aim behind this work is to propose a new solution that helps produce the composite video by merging the augmented video of the surgery site and the virtual hand of the remote expertise surgeon. The purpose of the proposed solution is to decrease the processing time and enhance the accuracy of merged video by decreasing the overlay and visualization error and remo…
Armstrong Atlantic State Univ.（阿姆斯特朗大西洋州立大学）；Ovidius University（奥维迪乌斯大学）
Surgeons must accomplish complex technical and intellectual tasks that can generate unexpected and serious challenges with little or no room for error. In the last decade, computer simulations have played an increasing role in surgical training, pre-operative planning, and biomedical research. Specifically, visuo-haptic simulations have been the focus of research to develop advanced e-Learning systems facilitating surgical training. The cost of haptic hardware was reduced through mass scale production and as haptics gained popularity in the gaming industry. Visuo-haptic simulations combine the…
Web-based Augmented Reality with Auto-Scaling and Real-Time Head Tracking towards Markerless Neurointerventional Preoperative Planning and Training of Head-mounted Robotic Needle Insertion
The Chinese University of Hong Kong（香港中文大学）；CUHK Shenzhen Research Institute（香港中文大学（深圳）研究院）
本文提出一种无标记Web增强现实应用，利用MediaPipe和React Three Fiber实现头部自动缩放与实时跟踪，用于神经介入术前规划和头戴式机器人针插入训练，成本低且支持协作。
With introduction of new technologies in the operating room like the da Vinci Surgical System, training surgeons to use them effectively and efficiently is crucial in the delivery of better patient care. Coaching by an expert surgeon is effective in teaching relevant technical skills, but current methods to deliver effective coaching are limited and not scalable. We present a virtual reality simulation-based framework for automated virtual coaching in surgical education. We implement our framework within the da Vinci Skills Simulator. We provide three coaching modes ranging from a hands-on tea…
University of British Columbia（不列颠哥伦比亚大学）；Technical University of Munich（慕尼黑工业大学）
Surgical instrument segmentation for robot-assisted surgery is needed for accurate instrument tracking and augmented reality overlays. Therefore, the topic has been the subject of a number of recent papers in the CAI community. Deep learning-based methods have shown state-of-the-art performance for surgical instrument segmentation, but their results depend on labelled data. However, labelled surgical data is of limited availability and is a bottleneck in surgical translation of these methods. In this paper, we demonstrate the limited generalizability of these methods on different datasets, inc…
Purpose Surgical simulations play an increasingly important role in surgeon education and developing algorithms that enable robots to perform surgical subtasks. To model anatomy, Finite Element Method (FEM) simulations have been held as the gold standard for calculating accurate soft-tissue deformation. Unfortunately, their accuracy is highly dependent on the simulation parameters, which can be difficult to obtain. Methods In this work, we investigate how live data acquired during any robotic endoscopic surgical procedure may be used to correct for inaccurate FEM simulation results. Since FEMs…
The Johns Hopkins University（约翰斯·霍普金斯大学）；Verb Surgical Inc.（Verb外科公司）；Ethicon Inc.（爱惜康公司）
Virtual Reality (VR) is rapidly increasing in popularity as a teaching tool. It allows for the creation of a highly immersive, three-dimensional virtual environment intended to simulate real-life environments. With more robots saturating the industry - from manufacturing to healthcare, there is a need to train end-users on how to set up, operate, tear down, and troubleshoot the robot. Even though VR has become widely used in training surgeons on the psychomotor skills associated with operating the robot, little research has been done to see how the benefits of VR could translate to teaching th…
Haptic Guidance and Haptic Error Amplification in a Virtual Surgical Robotic Training Environment
Stanford University（斯坦福大学）；Hospital for Special Surgery（特种外科医院）；University of Notre Dame（圣母大学）；University of Washington（华盛顿大学）；Ben-Gurion University of the Negev（内盖夫本-古里安大学）；Intuitive Surgical Inc.（直觉外科公司）；Case Western Reserve University（凯斯西储大学）
Toward Reliable AR-Guided Surgical Navigation: Interactive Deformation Modeling with Data-Driven Biomechanics and Prompts
Department of Computer Science and Engineering, The Chinese University of Hong Kong, HKSAR, China（计算机科学与工程系，香港中文大学（深圳））；Center of Smart Health, School of Nursing, The Hong Kong Polytechnic University, HKSAR, China（智能健康中心，香港理工大学）；Department of Hepatobiliary Surgery, The Third Affiliated Hospital of Southern Medical University, Guangzhou, China（肝胆外科部，南方医科大学第三附属医院）
Sano Centre for Computational Medicine（萨诺计算医学中心）；University of Amsterdam（阿姆斯特丹大学）；The Medical Centre of Postgraduate Education（研究生教育医学中心）
Spina Bifida (SB) is a birth defect developed during the early stage of pregnancy in which there is incomplete closing of the spine around the spinal cord. The growing interest in fetoscopic Spina-Bifida repair, which is performed in fetuses who are still in the pregnant uterus, prompts the need for appropriate training. The learning curve for such procedures is steep and requires excellent procedural skills. Computer-based virtual reality (VR) simulation systems offer a safe, cost-effective, and configurable training environment free from ethical and patient safety issues. However, to the bes…
Uncertainty-aware Self-supervised Learning for Cross-domain Technical Skill Assessment in Robot-assisted Surgery
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Scuola Superiore Sant’Anna（圣安娜高等研究院）；Politecnico di Milano（米兰理工大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
