Automatic surgical gesture recognition is fundamental for improving intelligence in robot-assisted surgery, such as conducting complicated tasks of surgery surveillance and skill evaluation. However, current methods treat each frame individually and produce the outcomes without effective consideration on future information. In this paper, we propose a framework based on reinforcement learning and tree search for joint surgical gesture segmentation and classification. An agent is trained to segment and classify the surgical video in a human-like manner whose direct decisions are re-considered by tree search appropriately. Our proposed tree search algorithm unites the outputs from two designed neural networks, i.e., policy and value network. With the integration of complementary information from distinct models, our framework is able to achieve the better performance than baseline methods using either of the neural networks. For an overall evaluation, our developed approach consistently outperforms the existing methods on the suturing task of JIGSAWS dataset in terms of accuracy, edit score and F1 score. Our study highlights the utilization of tree search to refine actions in reinforcement learning framework for surgical robotic applications.
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during R…
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery. With recent advancement in robot-assisted minimally invasive surgery, rich information including surgical videos and robotic kinematics can be recorded, which provide complementary knowledge for understanding surgical gestures. However, existing methods either solely adopt uni-modal data or directly concatenate multi-modal representations, which can not sufficiently exploit the informative correlations inherent in visual and kinematics data to boost gesture recogniti…
This chapter presents an overview of Interactive Machine Learning (IML) techniques applied to the analysis and design of musical gestures. We go through the main challenges and needs related to capturing, analysing, and applying IML techniques to human bodily gestures with the purpose of performing with sound synthesis systems. We discuss how different algorithms may be used to accomplish different tasks, including interacting with complex synthesis techniques and exploring interaction possibilities by means of Reinforcement Learning (RL) in an interaction paradigm we developed called Assisted…
The Chinese University of Hong Kong（香港中文大学）；The Johns Hopkins University（约翰斯·霍普金斯大学）
Robot-assisted surgery has become progressively more and more popular due to its clinical advantages. In the meanwhile, the artificial intelligence and augmented reality in robotic surgery are developing rapidly and receive lots of attention. However, current methods have not discussed the coherent integration of AI and AR in robotic surgery. In this paper, we develop a novel system by seamlessly merging artificial intelligence module and augmented reality visualization to automatically generate the surgical guidance for robotic surgery education. Specifically, we first leverage reinforcement …
Uncertainty-aware Self-supervised Learning for Cross-domain Technical Skill Assessment in Robot-assisted Surgery
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Scuola Superiore Sant’Anna（圣安娜高等研究院）；Politecnico di Milano（米兰理工大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Dynamic Robot-Assisted Surgery with Hierarchical Class-Incremental Semantic Segmentation
Department of Computer Science, University of Freiburg（弗赖堡大学计算机科学系）；Zuse School ELIZA（Zuse学校ELIZA）；NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montréal（神经Poly实验室，生物医学工程学院，蒙特利尔理工学院）；Mila - Québec AI Institute（魁北克人工智能研究所）；Collaborative Research Institute Intelligent Oncology (CRIION)（智能肿瘤学协作研究机构）
OSSAR: Towards Open-Set Surgical Activity Recognition in Robot-assisted Surgery
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Beijing Institute of Technology（北京理工大学）；Qilu Hospital of Shandong University（山东大学齐鲁医院）；Tongji University（同济大学）；University College London（伦敦大学学院）
Fiducial markers are commonly used in navigation assisted minimally invasive spine surgery (MISS) and they help transfer image coordinates into real world coordinates. In practice, these markers might be located outside the field-of-view (FOV), due to the limited detector sizes of C-arm cone-beam computed tomography (CBCT) systems used in intraoperative surgeries. As a consequence, reconstructed markers in CBCT volumes suffer from artifacts and have distorted shapes, which sets an obstacle for navigation. In this work, we propose two fiducial marker detection methods: direct detection from dis…
Monash University（莫纳什大学）；University of Haute-Alsace（上阿尔萨斯大学）
Over the past one hundred years, the classic teaching methodology of "see one, do one, teach one" has governed the surgical education systems worldwide. With the advent of Operation Room 2.0, recording video, kinematic and many other types of data during the surgery became an easy task, thus allowing artificial intelligence systems to be deployed and used in surgical and medical practice. Recently, surgical videos has been shown to provide a structure for peer coaching enabling novice trainees to learn from experienced surgeons by replaying those videos. However, the high inter-operator variab…
Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos
Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；University College London（伦敦大学学院）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences（Wellcome/EPSRC介入与外科科学中心）
SEDMamba: Enhancing Selective State Space Modelling with Bottleneck Mechanism and Fine-to-Coarse Temporal Fusion for Efficient Error Detection in Robot-Assisted Surgery
UCL Hawkes Institute（伦敦大学学院霍克斯研究所）；University College London（伦敦大学学院）；Griffin Institute（格里芬研究所）；Northwick Park and St Mark’s Hospital（诺斯威克公园与圣马克医院）
Technion – Israel Institute of Technology（以色列理工学院）；Stanford University（斯坦福大学）
In recent times there is a growing development of video based applications for surgical purposes. Part of these applications can work offline after the end of the procedure, other applications must react immediately. However, there are cases where the response should be done during the procedure but some delay is acceptable. In the literature, the online-offline performance gap is known. Our goal in this study was to learn the performance-delay trade-off and design an MS-TCN++-based algorithm that can utilize this trade-off. To this aim, we used our open surgery simulation data-set containing …
Department of Electrical and Computer Engineering, University of Arizona（电子与计算机工程系，亚利桑那大学）；Department of Computing and Data Science, University of California, Berkeley（计算与数据科学系，加州大学伯克利分校）；Quandary Peak Research（Quandary Peak研究公司）；Department of Systems and Industrial Engineering, University of Arizona（系统与工业工程系，亚利桑那大学）；College of Information Sciences, University of Arizona（信息科学学院，亚利桑那大学）
End to End AI System for Surgical Gesture Sequence Recognition and Clinical Outcome Prediction
Department of Computational Biomedicine, Center for Artificial Intelligence Research and Education（计算生物医学系，人工智能研究与教育中心）；Cedars Sinai Medical Center（Cedars Sinai 医疗中心）
Purpose: Recent developments in robotics and artificial intelligence (AI) have led to significant advances in healthcare technologies enhancing robot-assisted minimally invasive surgery (RAMIS) in some surgical specialties. However, current human-robot interfaces lack intuitive teleoperation and cannot mimic surgeon's hand/finger sensing and fine motion. These limitations make tele-operated robotic surgery not suitable for micro-surgery and difficult to learn for established surgeons. We report a pilot study showing an intuitive way of recording and mapping surgeon's gross hand motion and the …
This project concerns developing and validating an image guidance framework for application to a robotic-assisted fibular reduction in ankle fracture surgery. The aim is to produce and demonstrate proper functioning of software for automatic determination of directions for fibular repositioning with the ultimate goal of application to a robotic reduction procedure that can reduce the time and complexity of the procedure as well as provide the benefits of reduced error in ideal final fibular position, improved syndesmosis restoration and reduced incidence of post-traumatic osteoarthritis. The f…
