Modeling and automatically recognizing surgical activities are fundamental steps toward automation in surgery and play important roles in providing timely feedback to surgeons. Accurately recognizing surgical activities in video poses a challenging problem that requires an effective means of learning both spatial and temporal dynamics. Human gaze and visual saliency carry important information about visual attention and can be used to extract more relevant features that better reflect these spatial and temporal dynamics. In this study, we propose to use human gaze with a spatio-temporal attention mechanism for activity recognition in surgical videos. Our model consists of an I3D-based architecture, learns spatio-temporal features using 3D convolutions, as well as learns an attention map using human gaze as supervision. We evaluate our model on the Suturing task of JIGSAWS which is a publicly available surgical video understanding dataset. To our knowledge, we are the first to use human gaze for surgical activity recognition. Our results and ablation studies support the contribution of using human gaze to guide attention by outperforming state-of-the art models with an accuracy of 85.4%.
Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance
Surgical Artificial Intelligence Laboratory, Massachusetts General Hospital, MA, USA（马萨诸塞州总医院手术人工智能实验室）；Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, MA, USA（麻省理工学院计算机科学与人工智能实验室）；Department of Surgery, Duke University, NC, USA（达特茅斯大学外科系）
SurgFed: Language-guided Multi-Task Federated Learning for Surgical Video Understanding
National University of Singapore（国立新加坡大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）
Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions
Center for Humanoid Research, Korea Institute of Science and Technology（韩国科学技术研究院仿人机器人研究中心）；School of Electrical and Electronic Engineering, Yonsei University（延世大学电气与电子工程学院）
Advances in deep learning have enabled the development of models that have exhibited a remarkable tendency to recognize and even localize actions in videos. However, they tend to experience errors when faced with scenes or examples beyond their initial training environment. Hence, they fail to adapt to new domains without significant retraining with large amounts of annotated data. In this paper, we propose to overcome these limitations by moving to an open-world setting by decoupling the ideas of recognition and reasoning. Building upon the compositional representation offered by Grenander's …
Ophora: A Large-Scale Data-Driven Text-Guided Ophthalmic Surgical Video Generation Model
Shanghai Artificial Intelligence Laboratory, China（上海人工智能实验室）；Shanghai Jiao Tong University, China（上海交通大学）；Monash University, Australia（墨尔本大学）；East China Normal University, China（华东师范大学）；Eye Hospital, Wenzhou Medical University, China（温州医学院眼医院）；Shanghai Academy of Artificial Intelligence for Science, China（上海人工智能科学研究院）；Imperial College London, UK（伦敦帝国理工学院）；Shanghai Innovation Institute, China（上海创新研究院）
The increased availability and accuracy of eye-gaze tracking technology has sparked attention-related research in psychology, neuroscience, and, more recently, computer vision and artificial intelligence. The attention mechanism in artificial neural networks is known to improve learning tasks. However, no previous research has combined the network attention and human attention. This paper describes a gaze-guided class activation mapping (GG-CAM) method to directly regulate the formation of network attention based on expert radiologists' visual attention for the chest X-ray pathology classifica…
Information Science and Technology College, Dalian Maritime University（大连海事大学信息科学技术学院）
Recently, the progress of learning-by-synthesis has proposed a training model for synthetic images, which can effectively reduce the cost of human and material resources. However, due to the different distribution of synthetic images compared to real images, the desired performance cannot be achieved. Real images consist of multiple forms of light orientation, while synthetic images consist of a uniform light orientation. These features are considered to be characteristic of outdoor and indoor scenes, respectively. To solve this problem, the previous method learned a model to improve the reali…
Rochester Institute of Technology（罗切斯特理工学院）；University of Washington（华盛顿大学）；Seattle Children’s Hospital（西雅图儿童医院）
Abstract. Purpose: This paper presents a scheme for generating virtual intraoperative CT scans in order to improve surgical completeness in Endoscopic Sinus Surgeries (ESS). Approach: The work presents three methods, the tip motion-based, the tip trajectory-based, and the instrument based, along with non-parametric smoothing and Gaussian Process Regression, for virtual intraoperative CT generation. Results: The proposed methods studied and compared on ESS performed on cadavers. Surgical results show all three methods improve the Dice Similarity Coefficients > 86%, with F-score > 92% and precis…
EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis
School of Computing, University of Georgia（佐治亚大学计算学院）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Georgia（佐治亚大学）；Department of Physics and Astronomy（物理与天文学系）
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（杰梅利大学医院基金会）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡大学医院-图像引导手术研究所）；Institute for Research against Digestive Cancer (IRCAD)（消化癌症研究所）
Minimally invasive image-guided surgery heavily relies on vision. Deep learning models for surgical video analysis could therefore support visual tasks such as assessing the critical view of safety (CVS) in laparoscopic cholecystectomy (LC), potentially contributing to surgical safety and efficiency. However, the performance, reliability and reproducibility of such models are deeply dependent on the quality of data and annotations used in their development. Here, we present a protocol, checklists, and visual examples to promote consistent annotation of hepatocystic anatomy and CVS criteria. We…
GlanceSeg: Real-time microaneurysm lesion segmentation with gaze-map-guided foundation model for early detection of diabetic retinopathy
Southern University of Science and Technology（南方科技大学）；Wenzhou Medical University（温州医科大学）；Intelligent Vision Plus Technology Co., Ltd.（深圳市智能视觉加科技有限公司）；The Chinese University of Hong Kong（香港中文大学）
Endo-NeRF++: Uncertainty-Aware Neural Rendering with Multi-Resolution Hash Encoding for Dynamic Surgical Scene Reconstruction
Finnish Center for Artificial Intelligence (FCAI)（芬兰人工智能中心）；ELLIS Unit Helsinki（欧洲学习与智能系统研究所赫尔辛基分部）；University of Helsinki（赫尔辛基大学）；Institute for Urban Studies (Urbaria)（城市研究所）；Institute of Sustainability Science (HELSUS)（可持续发展科学研究所）
Attention-Aware Laparoscopic Image Desmoking Network with Lightness Embedding and Hybrid Guided Embedding
School of Life Science and Technology, Harbin Institute of Technology（哈尔滨工业大学生命科学与技术学院）；State Key Laboratory of Robotics and System, Harbin Institute of Technology（哈尔滨工业大学机器人技术与系统国家重点实验室）；State Key Laboratory of Robotics, Shenyang Institute of Automation（沈阳自动化研究所机器人学国家重点实验室）
Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；Georgia Tech Research Institute（佐治亚理工研究院）
This paper presents a method of navigating a surgical robot beyond the resolution of magnetic resonance imaging (MRI) by using a resolution enhancement technique enabled by high-precision piezoelectric actuation. The surgical robot was specifically designed for injecting stem cells into the spinal cord. This particular therapy can be performed in a shorter time by using a MRI-compatible robotic platform than by using a manual needle positioning platform. Imaging resolution of fiducial markers attached to the needle guide tubing was enhanced by reconstructing a high-resolution image from multip…
Intraoperative assessment of tissue can be guided through fluorescence imaging which involves systemic dosing with a fluorophore and subsequent examination of the tissue region of interest with a near-infrared camera. This typically involves administering indocyanine green (ICG) hours or even days before surgery and intraoperative visualization at the time predicted for steady-state signal-to-background status. Here, we describe our efforts to capture and utilize the information contained in the first few minutes after ICG administration from the perspective of both signal processing and surgi…
