Conventional therapy approaches limit surgeons' dexterity control due to limited field-of-view. With the advent of robot-assisted surgery, there has been a paradigm shift in medical technology for minimally invasive surgery. However, it is very challenging to track the position of the surgical instruments in a surgical scene, and accurate detection & identification of surgical tools is paramount. Deep learning-based semantic segmentation in frames of surgery videos has the potential to facilitate this task. In this work, we modify the U-Net architecture named U-NetPlus, by introducing a pre-trained encoder and re-design the decoder part, by replacing the transposed convolution operation with an upsampling operation based on nearest-neighbor (NN) interpolation. To further improve performance, we also employ a very fast and flexible data augmentation technique. We trained the framework on 8 x 225 frame sequences of robotic surgical videos, available through the MICCAI 2017 EndoVis Challenge dataset and tested it on 8 x 75 frame and 2 x 300 frame videos. Using our U-NetPlus architecture, we report a 90.20% DICE for binary segmentation, 76.26% DICE for instrument part segmentation, and 46.07% for instrument type (i.e., all instruments) segmentation, outperforming the results of previous techniques implemented and tested on these data.
A spatio-temporal network for video semantic segmentation in surgical videos
Medtronic Digital Surgery（美敦力数字外科）；Wellcome/EPSRC Centre for Internventional and Surgical Sciences, University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科学中心）
Surgical-DeSAM: Decoupling SAM for Instrument Segmentation in Robotic Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（Wellcome/EPSRC介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；Dept. of Computer Science, University College London（伦敦大学学院计算机科学系）
Bournemouth University（伯恩茅斯大学）；Technical University of Munich（慕尼黑工业大学）
Automatic surgical instruction generation is a prerequisite towards intra-operative context-aware surgical assistance. However, generating instructions from surgical scenes is challenging, as it requires jointly understanding the surgical activity of current view and modelling relationships between visual information and textual description. Inspired by the neural machine translation and imaging captioning tasks in open domain, we introduce a transformer-backboned encoder-decoder network with self-critical reinforcement learning to generate instructions from surgical images. We evaluate the ef…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Weill Cornell Graduate School for Medical Sciences（威尔康奈尔医学研究生院）
Pathologic analysis of surgical excision specimens for breast carcinoma is important to evaluate the completeness of surgical excision and has implications for future treatment. This analysis is performed manually by pathologists reviewing histologic slides prepared from formalin-fixed tissue. In this paper, we present Deep Multi-Magnification Network trained by partial annotation for automated multi-class tissue segmentation by a set of patches from multiple magnifications in digitized whole slide images. Our proposed architecture with multi-encoder, multi-decoder, and multi-concatenation out…
NVIDIA（英伟达）；Technische Universität München（慕尼黑工业大学）
Recent neural-network-based architectures for image segmentation make extensive usage of feature forwarding mechanisms to integrate information from multiple scales. Although yielding good results, even deeper architectures and alternative methods for feature fusion at different resolutions have been scarcely investigated for medical applications. In this work we propose to implement segmentation via an encoder-decoder architecture which differs from any other previously published method since (i) it employs a very deep architecture based on residual learning and (ii) combines features via a c…
State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统管理与控制国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
The real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, it is still a challenging task to implement deep learning models to do real-time segmentation for surgical instruments due to their high computational costs and slow inference speed. In this paper, we propose an attention-guided lightweight network (LWANet), which can segment surgical instruments in real-time. LWANet adopts encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2, and the decoder consists of depthwise separable convolution, attention fusion …
Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery
Department of Biomedical Engineering, Eindhoven University of Technology（埃因霍温理工大学生物医学工程系）；Department of Electrical Engineering, Eindhoven University of Technology（埃因霍温理工大学电气工程系）；Department of Surgery, University Medical Center Utrecht（乌得勒支大学医学中心外科）
Cancer of the brain is deadly and requires careful surgical segmentation. The brain tumors were segmented using U-Net using a Convolutional Neural Network (CNN). When looking for overlaps of necrotic, edematous, growing, and healthy tissue, it might be hard to get relevant information from the images. The 2D U-Net network was improved and trained with the BraTS datasets to find these four areas. U-Net can set up many encoder and decoder routes that can be used to get information from images that can be used in different ways. To reduce computational time, we use image segmentation to exclude i…
SimulaMet（西穆拉大都会实验室）；UiT The Arctic University of Norway（挪威北极大学）；Oslo Metropolitan University（奥斯陆城市大学）；University of Oxford（牛津大学）；NIHR Oxford Biomedical Research Centre（英国国家健康研究所牛津生物医学研究中心）
Minimally invasive surgery is a surgical intervention used to examine the organs inside the abdomen and has been widely used due to its effectiveness over open surgery. Due to the hardware improvements such as high definition cameras, this procedure has significantly improved and new software methods have demonstrated potential for computer-assisted procedures. However, there exists challenges and requirements to improve detection and tracking of the position of the instruments during these surgical procedures. To this end, we evaluate and compare some popular deep learning methods that can be…
SAGS: Self-Adaptive Alias-Free Gaussian Splatting for Dynamic Surgical Endoscopic Reconstruction
University of Technology Sydney, Australia（悉尼技术大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China（深圳先进技术研究所）；Shenyang Institute of Automation, Chinese Academy of Sciences, China（沈阳自动化研究所）
Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance
Surgical Artificial Intelligence Laboratory, Massachusetts General Hospital, MA, USA（马萨诸塞州总医院手术人工智能实验室）；Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, MA, USA（麻省理工学院计算机科学与人工智能实验室）；Department of Surgery, Duke University, NC, USA（达特茅斯大学外科系）
Shenzhen University（深圳大学）；University of Sydney（悉尼大学）；Northwestern Polytechnical University（西北工业大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Assessing the location and extent of lesions caused by chronic stroke is critical for medical diagnosis, surgical planning, and prognosis. In recent years, with the rapid development of 2D and 3D convolutional neural networks (CNN), the encoder-decoder structure has shown great potential in the field of medical image segmentation. However, the 2D CNN ignores the 3D information of medical images, while the 3D CNN suffers from high computational resource demands. This paper proposes a new architecture called dimension-fusion-UNet (D-UNet), which combines 2D and 3D convolution innovatively in the…
Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心）；National University of Singapore（新加坡国立大学）
Surgical scene understanding and multi-tasking learning are crucial for image-guided robotic surgery. Training a real-time robotic system for the detection and segmentation of high-resolution images provides a challenging problem with the limited computational resource. The perception drawn can be applied in effective real-time feedback, surgical skill assessment, and human-robot collaborative surgeries to enhance surgical outcomes. For this purpose, we develop a novel end-to-end trainable real-time Multi-Task Learning (MTL) model with weight-shared encoder and task-aware detection and segment…
Towards Patient-Specific Deformable Registration in Laparoscopic Surgery
Biomedical Robotics Lab, Istituto Italiano di Tecnologia, Genoa, Italy（生物医学机器人实验室，意大利理工学院，热那亚）；Department of Computer Science, Bioengineering, Robotics and Systems Engineering (DIBRIS), University of Genoa, Italy（计算机科学、生物工程、机器人与系统工程系（DIBRIS），热那亚大学，意大利）；Harvard Medical School, Brigham and Women’s Hospital, Boston, USA（哈佛医学院，布里洛妇女医院，美国波士顿）
LapFM: A Laparoscopic Segmentation Foundation Model via Hierarchical Concept Evolving Pre-training
School of Computer Science, University of Lincoln, UK（英国林肯大学计算机科学学院）；University of Nottingham, UK（英国诺丁汉大学）；University of Nottingham Ningbo China, China（中国宁波诺丁汉大学）；University of Strasbourg, France（法国斯特拉斯堡大学）；Technical University of Munich, Germany（德国慕尼黑技术大学）；Graduate School of Information, Production and Systems, Waseda University, Japan（日本早稻田大学信息、生产与系统研究生院）；Department of Gastrointestinal Surgery, The Second Qilu Hospital, Shandong University, China（中国山东大学第二齐鲁医院胃肠外科）；School of Engineering and Physical Science, University of Lincoln, Lincoln LN6 7TS, UK（英国林肯大学工程与物理科学学院）；Yale University, New Haven, CT 06510, USA（美国耶鲁大学）
A Total Variation Regularized Framework for Epilepsy-Related MRI Image Segmentation
Department of Computer Engineering, Modeling, Electronics and Systems (DIMES), University of Calabria, Rende 87036, Italy（计算机工程、建模、电子与系统系（DIMES），卡拉布里亚大学，Rende 87036，意大利）
DualSwinUnet++: An Enhanced Swin-Unet Architecture With Dual Decoders For PTMC Segmentation
Department of Mechanical and Mechatronics Engineering（机械与机电工程系）；University of Waterloo（滑铁卢大学）；Department of Radiology Withsim Clinic Seongnam, Korea（Seongnam诊所放射科）
SimulaMet（西穆拉梅特研究所）；UIT The Arctic University of Norway（挪威北极大学）；University of Oxford（牛津大学）；Oslo Metropolitan University（奥斯陆城市大学）
Colonoscopy is the gold standard for examination and detection of colorectal polyps. Localization and delineation of polyps can play a vital role in treatment (e.g., surgical planning) and prognostic decision making. Polyp segmentation can provide detailed boundary information for clinical analysis. Convolutional neural networks have improved the performance in colonoscopy. However, polyps usually possess various challenges, such as intra-and inter-class variation and noise. While manual labeling for polyp assessment requires time from experts and is prone to human error (e.g., missed lesions)…
NUS Graduate School for Integrative Sciences and Engineering (NGS), National University of Singapore（新加坡国立大学综合科学与工程研究生院）；Dept. of Biomedical Engineering, National University of Singapore（新加坡国立大学生物医学工程系）；Dept. of Instrumentation and Control Engineering, NIT Trichy（印度国立理工学院蒂鲁奇拉帕利分校仪器与控制工程系）；Dept. of Otolaryngology, Singapore General Hospital（新加坡中央医院耳鼻喉科）
Representation learning of the task-oriented attention while tracking instrument holds vast potential in image-guided robotic surgery. Incorporating cognitive ability to automate the camera control enables the surgeon to concentrate more on dealing with surgical instruments. The objective is to reduce the operation time and facilitate the surgery for both surgeons and patients. We propose an end-to-end trainable Spatio-Temporal Multi-Task Learning (ST-MTL) model with a shared encoder and spatio-temporal decoders for the real-time surgical instrument segmentation and task-oriented saliency dete…
An Efficient Dual-Line Decoder Network with Multi-Scale Convolutional Attention for Multi-organ Segmentation
Institute of Information and Communication Technology, Bangladesh University of Engineering and Technology（信息与通信技术研究所，孟加拉国工程与技术大学）；Computer Science and Engineering, Green university of Bangladesh（计算机科学与工程，绿色大学）；Computer Science, Marquette University（计算机科学，马基特大学）；New College of Interdisciplinary Arts and Sciences, Arizona State University（跨学科艺术与科学新学院，亚利桑那州立大学）；Electrical and Computer Engineering, The University of Texas at Austin（电气与计算机工程，德克萨斯大学奥斯汀分校）
NUS Graduate School for Integrative Sciences and Engineering (NGS), National University of Singapore（新加坡国立大学NUS综合科学与工程研究生院（NGS））；National University of Singapore（新加坡国立大学）；University of Michigan-Joint Institute, Shanghai Jiao Tong University（上海交通大学密歇根联合学院）
Directing of the task-specific attention while tracking instrument in surgery holds great potential in robot-assisted intervention. For this purpose, we propose an end-to-end trainable multitask learning (MTL) model for real-time surgical instrument segmentation and attention prediction. Our model is designed with a weight-shared encoder and two task-oriented decoders and optimized for the joint tasks. We introduce batch-Wasserstein (bW) loss and construct a soft attention module to refine the distinctive visual region for efficient saliency learning. For multitask optimization, it is always c…
HRVVS: A High-resolution Video Vasculature Segmentation Network via Hierarchical Autoregressive Residual Priors
The Hong Kong University of Science and Technology (Guangzhou), China（香港科学与技术大学（广州））；Southern Medical University, Guangzhou, China（南方医科大学）；The Hong Kong University of Science and Technology, Hong Kong, China（香港科学与技术大学）
University of Tehran（德黑兰大学）；Nara Institute of Science and Technology（奈良科学技术研究所）；Osaka University Graduate School of Medicine（大阪大学医学院）
With the increasing usage of radiograph images as a most common medical imaging system for diagnosis, treatment planning, and clinical studies, it is increasingly becoming a vital factor to use machine learning-based systems to provide reliable information for surgical pre-planning. Segmentation of pelvic bone in radiograph images is a critical preprocessing step for some applications such as automatic pose estimation and disease detection. However, the encoder-decoder style network known as U-Net has demonstrated limited results due to the challenging complexity of the pelvic shapes, especial…
SINTEF（挪威科技工业研究院）；Bristol Royal Hospital for Children（布里斯托尔皇家儿童医院）；St. Olavs hospital（圣奥拉夫医院）
Automatic and consistent meningioma segmentation in T1-weighted MRI volumes and corresponding volumetric assessment is of use for diagnosis, treatment planning, and tumor growth evaluation. In this paper, we optimized the segmentation and processing speed performances using a large number of both surgically treated meningiomas and untreated meningiomas followed at the outpatient clinic. We studied two different 3D neural network architectures: (i) a simple encoder-decoder similar to a 3D U-Net, and (ii) a lightweight multi-scale architecture (PLS-Net). In addition, we studied the impact of dif…
SINTEF（挪威科技工业研究院）；Bristol Royal Hospital for Children（布里斯托尔皇家儿童医院）；St. Olavs hospital（圣奥拉夫斯医院）
Meningiomas are the most common type of primary brain tumor, accounting for approximately 30% of all brain tumors. A substantial number of these tumors are never surgically removed but rather monitored over time. Automatic and precise meningioma segmentation is therefore beneficial to enable reliable growth estimation and patient-specific treatment planning. In this study, we propose the inclusion of attention mechanisms over a U-Net architecture: (i) Attention-gated U-Net (AGUNet) and (ii) Dual Attention U-Net (DAUNet), using a 3D MRI volume as input. Attention has the potential to leverage t…
