Image segmentation is often performed on medical images for identifying diseases in clinical evaluation. Hence it has become one of the major research areas. Conventional image segmentation techniques are unable to provide satisfactory segmentation results for medical images as they contain irregularities. They need to be pre-processed before segmentation. In order to obtain the most suitable method for medical image segmentation, we propose a two stage algorithm. The first stage automatically generates a binary marker image of the region of interest using mathematical morphology. This marker serves as the mask image for the second stage which uses GrabCut on the input image thus resulting in an efficient segmented result. The obtained result can be further refined by user interaction which can be done using the Graphical User Interface (GUI). Experimental results show that the proposed method is accurate and provides satisfactory segmentation results with minimum user interaction on medical as well as natural images.
This paper presents an interactive approach for multi-class segmentation of aerial images. Precisely, it is based on a deep neural network which exploits both RGB images and annotations. Starting from an initial output based on the image only, our network then interactively refines this segmentation map using a concatenation of the image and user annotations. Importantly, user annotations modify the inputs of the network - not its weights - enabling a fast and smooth process. Through experiments on two public aerial datasets, we show that user annotations are extremely rewarding: each click co…
Brigham Young University（杨百翰大学）；Adobe Research（Adobe 研究院）
As deep neural networks become the state-of-the-art approach in the field of computer vision for dense prediction tasks, many methods have been developed for automatic estimation of the target outputs given the visual inputs. Although the estimation accuracy of the proposed automatic methods continues to improve, interactive refinement is oftentimes necessary for further correction. Recently, feature backpropagating refinement scheme (f-BRS) has been proposed for the task of interactive segmentation, which enables efficient optimization of a small set of auxiliary variables inserted into the p…
Deep learning techniques have successfully been employed in numerous computer vision tasks including image segmentation. The techniques have also been applied to medical image segmentation, one of the most critical tasks in computer-aided diagnosis. Compared with natural images, the medical image is a gray-scale image with low-contrast (even with some invisible parts). Because some organs have similar intensity and texture with neighboring organs, there is usually a need to refine automatic segmentation results. In this paper, we propose an interactive deep refinement framework to improve the …
Clore: Interactive Pathology Image Segmentation with Click-based Local Refinement
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）
East China Normal University（华东师范大学）；Huashan Hospital, Fudan University（复旦大学附属华山医院）
Medical image segmentation is one of the fundamental problems for artificial intelligence-based clinical decision systems. Current automatic medical image segmentation methods are often failed to meet clinical requirements. As such, a series of interactive segmentation algorithms are proposed to utilize expert correction information. However, existing methods suffer from some segmentation refining failure problems after long-term interactions and some cost problems from expert annotation, which hinder clinical applications. This paper proposes an interactive segmentation framework, called inte…
Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline
Shanghai AI Laboratory（上海人工智能实验室）；Sichuan University（四川大学）；Monash University（莫纳什大学）；East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）；Xinjiang University（新疆大学）
Interactive 3D Medical Image Segmentation with SAM 2
East China Normal University（华东师范大学）；Tongji University（同济大学）；Shanghai United Imaging Healthcare Advanced Technology Research Institute Co., Ltd.（上海联影医疗科技先进技术研究院有限公司）
本文提出将 3D 医学切片视为视频帧、利用 SAM 2 零样本传播单帧标注的交互式分割流程，实验表明其可在特定场景缩小与监督方法差距并降低标注成本。
Dynamic Prompt Generation for Interactive 3D Medical Image Segmentation Training
Medical Physics, Department of Diagnostic and Interventional Radiology, Medical Center—University of Freiburg, Faculty of Medicine, University of Freiburg（医学物理，诊断与介入放射科，弗赖堡大学医学中心，医学院，弗赖堡大学）；University of Freiburg（弗赖堡大学）；ELLIS Institute Tübingen（图宾根ELLIS研究所）
Peking University（北京大学）；Deepwise AI Lab（深睿医疗人工智能实验室）；Microsoft Research Asia（微软亚洲研究院）；The University of Hong Kong（香港大学）
As an important upstream task for many medical applications, supervised landmark localization still requires non-negligible annotation costs to achieve desirable performance. Besides, due to cumbersome collection procedures, the limited size of medical landmark datasets impacts the effectiveness of large-scale self-supervised pre-training methods. To address these challenges, we propose a two-stage framework for one-shot medical landmark localization, which first infers landmarks by unsupervised registration from the labeled exemplar to unlabeled targets, and then utilizes these noisy pseudo l…
Nanjing University（南京大学）；University of Wollongong（伍伦贡大学）；Nanjing Normal University（南京师范大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Due to low tissue contrast, irregular object appearance, and unpredictable location variation, segmenting the objects from different medical imaging modalities (e.g., CT, MR) is considered as an important yet challenging task. In this paper, we present a novel method for interactive medical image segmentation with the following merits. (1) Our design is fundamentally different from previous pure patch-based and image-based segmentation methods. We observe that during delineation, the physician repeatedly check the inside-outside intensity changing to determine the boundary, which indicates tha…
Shanghai Jiao Tong University（上海交通大学）；East China Normal University（华东师范大学）
Existing automatic 3D image segmentation methods usually fail to meet the clinic use. Many studies have explored an interactive strategy to improve the image segmentation performance by iteratively incorporating user hints. However, the dynamic process for successive interactions is largely ignored. We here propose to model the dynamic process of iterative interactive image segmentation as a Markov decision process (MDP) and solve it with reinforcement learning (RL). Unfortunately, it is intractable to use single-agent RL for voxel-wise prediction due to the large exploration space. To reduce …
Lightweight Method for Interactive 3D Medical Image Segmentation with Multi-Round Result Fusion
Aerospace Center Hospital（航天中心医院）；Yizhun Medical AI Co., Ltd（医准智能医疗科技有限公司）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Southern University of Science and Technology（南方科技大学）；Beijing Institute of Technology（北京理工大学）
Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation
School of Instrument Science and Engineering, Southeast University（东南大学仪器科学与工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Sichuan University-Pittsburgh Institute, Sichuan University（四川大学匹兹堡学院）；Department of Biomedical Engineering, Case Western Reserve University（凯斯西储大学生物医学工程系）
University College London（伦敦大学学院）；KU Leuven（鲁汶大学）；University Hospitals KU Leuven（鲁汶大学医院）
Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes. To address these problems, we propose a novel deep learning-based framework for interactive segmentation by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine-tuning to make a CNN m…
MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning
Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学）；Hunyuan Group, Tencent（腾讯洪音集团）；Institute of Automation, the Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；Dalian University of Technology, Dalian, China（大连理工大学）；Stanford University, Stanford, USA（斯坦福大学）
Temporally-Extended Prompts Optimization for SAM in Interactive Medical Image Segmentation
East China Normal University（华东师范大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；Shanghai Jiao Tong University（上海交通大学）
ETH Zurich（苏黎世联邦理工学院）；Beijing Institute of Technology（北京理工大学）
Despite recent progress of automatic medical image segmentation techniques, fully automatic results usually fail to meet the clinical use and typically require further refinement. In this work, we propose a quality-aware memory network for interactive segmentation of 3D medical images. Provided by user guidance on an arbitrary slice, an interaction network is firstly employed to obtain an initial 2D segmentation. The quality-aware memory network subsequently propagates the initial segmentation estimation bidirectionally over the entire volume. Subsequent refinement based on additional user gui…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
We propose iSegFormer, a memory-efficient transformer that combines a Swin transformer with a lightweight multilayer perceptron (MLP) decoder. With the efficient Swin transformer blocks for hierarchical self-attention and the simple MLP decoder for aggregating both local and global attention, iSegFormer learns powerful representations while achieving high computational efficiencies. Specifically, we apply iSegFormer to interactive 3D medical image segmentation.
Efficient Universal Models for Medical Image Segmentation via Weakly Supervised In-Context Learning
Harbin Institute of Technology at Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；University Hospital Tübingen（图宾根大学医院）；German Center for Mental Health（德国心理健康中心）
Universal Topology Refinement for Medical Image Segmentation with Polynomial Feature Synthesis
Imperial College London（帝国理工学院）；University of Oxford（牛津大学）；Klinikum rechts der Isar, Technical University of Munich（慕尼黑工业大学右侧医院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
Medical image segmentation, the task of partitioning an image into meaningful parts, is an important step toward automating medical image analysis and is at the crux of a variety of medical imaging applications, such as computer aided diagnosis, therapy planning and delivery, and computer aided interventions. However, the existence of noise, low contrast and objects' complexity in medical images are critical obstacles that stand in the way of achieving an ideal segmentation system. Incorporating prior knowledge into image segmentation algorithms has proven useful for obtaining more accurate an…
Explainable Continuous-Time Mask Refinement with Local Self-Similarity Priors for Medical Image Segmentation
School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India（计算机工程学院，印度克里希纳加尔工业技术学院，布巴内斯瓦尔，印度）；AmygdalaAI-India Lab, Bhubaneswar, India（AmygdalaAI-印度实验室，布巴内斯瓦尔，印度）
Image segmentation is a fundamental problem in computational vision and medical imaging. Designing a generic, automated method that works for various objects and imaging modalities is a formidable task. Instead of proposing a new specific segmentation algorithm, we present a general design principle on how to integrate user interactions from the perspective of feedback control theory. Impulsive control and Lyapunov stability analysis are employed to design and analyze an interactive segmentation system. Then stabilization conditions are derived to guide algorithm design. Finally, the effective…
Multi-atlas segmentation (MAS) is a popular image segmentation technique for medical images. In this work, we improve the performance of MAS by correcting registration errors before label fusion. Specifically, we use a volumetric displacement field to refine registrations based on image anatomical appearance and predicted labels. We show the influence of the initial spatial alignment as well as the beneficial effect of using label information for MAS performance. Experiments demonstrate that the proposed refinement approach improves MAS performance on a 3D magnetic resonance dataset of the kne…
Leipzig University（莱比锡大学）；IfADo - Leibniz Research Centre for Working Environment and Human Factors（莱布尼茨工作环境与人因研究中心）；INRIA Paris（法国国家信息与自动化研究所巴黎中心）；Sorbonne Université（索邦大学）
We present a novel approach that combines machine learning based interactive image segmentation using supervoxels with a clustering method for the automated identification of similarly colored images in large data sets which enables a guided reuse of classifiers. Our approach solves the problem of significant color variability prevalent and often unavoidable in biological and medical images which typically leads to deteriorated segmentation and quantification accuracy thereby greatly reducing the necessary training effort. This increase in efficiency facilitates the quantification of much larg…
University of Electronic Science and Technology of China（电子科技大学）；King’s College London（伦敦国王学院）；University Hospitals KU Leuven（鲁汶大学附属医院）；University College London（伦敦大学学院）；SenseTime Research（商汤科技研究院）
Segmentation of the fetal brain from stacks of motion-corrupted fetal MRI slices is important for motion correction and high-resolution volume reconstruction. Although Convolutional Neural Networks (CNNs) have been widely used for automatic segmentation of the fetal brain, their results may still benefit from interactive refinement for challenging slices. To improve the efficiency of interactive refinement process, we propose an Uncertainty-Guided Interactive Refinement (UGIR) framework. We first propose a grouped convolution-based CNN to obtain multiple automatic segmentation predictions with…
Deep neural networks have become a mainstream approach to interactive segmentation. As we show in our experiments, while for some images a trained network provides accurate segmentation result with just a few clicks, for some unknown objects it cannot achieve satisfactory result even with a large amount of user input. Recently proposed backpropagating refinement (BRS) scheme introduces an optimization problem for interactive segmentation that results in significantly better performance for the hard cases. At the same time, BRS requires running forward and backward pass through a deep network s…
International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）；Indian Institute of Technology, Delhi（德里印度理工学院）
Semantic segmentation of medical images is an essential first step in computer-aided diagnosis systems for many applications. However, given many disparate imaging modalities and inherent variations in the patient data, it is difficult to consistently achieve high accuracy using modern deep neural networks (DNNs). This has led researchers to propose interactive image segmentation techniques where a medical expert can interactively correct the output of a DNN to the desired accuracy. However, these techniques often need separate training data with the associated human interactions, and do not g…
We present a meta-learning framework for interactive medical image registration. Our proposed framework comprises three components: a learning-based medical image registration algorithm, a form of user interaction that refines registration at inference, and a meta-learning protocol that learns a rapidly adaptable network initialization. This paper describes a specific algorithm that implements the registration, interaction and meta-learning protocol for our exemplar clinical application: registration of magnetic resonance (MR) imaging to interactively acquired, sparsely-sampled transrectal ult…
Interactive Medical-SAM2 GUI: A Napari-based semi-automatic annotation tool for medical images
Department of Biomechatronic Engineering, Sungkyunkwan University（生物机电工程系，全州大学）；Department of Neurosurgery, Seoul National University Hospital, Seoul National University College of Medicine（神经外科，首尔国立大学医院，首尔国立大学医学院）
We propose a new generative adversarial architecture to mitigate imbalance data problem in medical image semantic segmentation where the majority of pixels belongs to a healthy region and few belong to lesion or non-health region. A model trained with imbalanced data tends to bias toward healthy data which is not desired in clinical applications and predicted outputs by these networks have high precision and low sensitivity. We propose a new conditional generative refinement network with three components: a generative, a discriminative, and a refinement network to mitigate unbalanced data prob…
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
The goal of this paper is to interactively refine the automatic segmentation on challenging structures that fall behind human performance, either due to the scarcity of available annotations or the difficulty nature of the problem itself, for example, on segmenting cancer or small organs. Specifically, we propose a novel Transformer-based architecture for Interactive Segmentation (TIS), that treats the refinement task as a procedure for grouping pixels with similar features to those clicks given by the end users. Our proposed architecture is composed of Transformer Decoder variants, which natu…
SLIP: Segmentation with Low-latency Interactive Prompting for 3D Medical Images
IRCAD France（法国国际消化道疾病研究中心）；Université de Strasbourg, CNRS, ICube（斯特拉斯堡大学、法国国家科学研究中心、ICube实验室）；IRCCS Humanitas Research Hospital（圣心研究医院）；Humanitas University（圣心大学）；Fondazione Policlinico Universitario A. Gemelli, Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS)（圣心综合大学医院）
University of Electronic Science and Technology of China（电子科技大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；University Hospitals Leuven（鲁汶大学医院）；University College London（伦敦大学学院）；King’s College London（伦敦国王学院）
Segmentation of organs or lesions from medical images plays an essential role in many clinical applications such as diagnosis and treatment planning. Though Convolutional Neural Networks (CNN) have achieved the state-of-the-art performance for automatic segmentation, they are often limited by the lack of clinically acceptable accuracy and robustness in complex cases. Therefore, interactive segmentation is a practical alternative to these methods. However, traditional interactive segmentation methods require a large amount of user interactions, and recently proposed CNN-based interactive segmen…
Large language models (LLMs) have recently demonstrated their potential in clinical applications, providing valuable medical knowledge and advice. For example, a large dialog LLM like ChatGPT has successfully passed part of the US medical licensing exam. However, LLMs currently have difficulty processing images, making it challenging to interpret information from medical images, which are rich in information that supports clinical decisions. On the other hand, computer-aided diagnosis (CAD) networks for medical images have seen significant success in the medical field by using advanced deep-le…
