A major focus of clinical imaging workflow is disease diagnosis and management, leading to medical imaging datasets strongly tied to specific clinical objectives. This scenario has led to the prevailing practice of developing task-specific segmentation models, without gaining insights from widespread imaging cohorts. Inspired by the training program of medical radiology residents, we propose a shift towards universal medical image segmentation, a paradigm aiming to build medical image understanding foundation models by leveraging the diversity and commonality across clinical targets, body regions, and imaging modalities. Towards this goal, we develop Hermes, a novel context-prior learning approach to address the challenges of data heterogeneity and annotation differences in medical image segmentation. In a large collection of eleven diverse datasets (2,438 3D images) across five modalities (CT, PET, T1, T2 and cine MRI) and multiple body regions, we demonstrate the merit of the universal paradigm over the traditional paradigm on addressing multiple tasks within a single model. By exploiting the synergy across tasks, Hermes achieves state-of-the-art performance on all testing datasets and shows superior model scalability. Results on two additional datasets reveals Hermes' strong performance for transfer learning, incremental learning, and generalization to downstream tasks. Hermes's learned priors demonstrate an appealing trait to reflect the intricate relations among tasks and modalities, which aligns with the established anatomical and imaging principles in radiology. The code is available: https://github.com/yhygao/universal-medical-image-segmentation.
CE-Net: Context Encoder Network for 2D Medical Image Segmentation
School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；Cixi Institute of Biomedical Engineering, Chinese Academy of Sciences（中国科学院慈溪生物医学工程研究所）；Inception Institute of Artificial Intelligence（Inception人工智能研究所）；Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore（新加坡科技研究局信息通信研究院）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）
SimTxtSeg: Weakly-Supervised Medical Image Segmentation with Simple Text Cues
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China（教育部新一代人工智能技术及交叉应用重点实验室）；Nanjing University of Science and Technology（南京理工大学）；Northwestern Polytechnical University（西北工业大学）
UiT The Arctic University of Norway（特罗姆瑟大学（挪威北极圈大学））
Recent work has shown that label-efficient few-shot learning through self-supervision can achieve promising medical image segmentation results. However, few-shot segmentation models typically rely on prototype representations of the semantic classes, resulting in a loss of local information that can degrade performance. This is particularly problematic for the typically large and highly heterogeneous background class in medical image segmentation problems. Previous works have attempted to address this issue by learning additional prototypes for each class, but since the prototypes are based on…
University of Electronic Science and Technology of China（电子科技大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）；West China Hospital, Sichuan University（四川大学华西医院）
Background and Objective: Open-source deep learning toolkits are one of the driving forces for developing medical image segmentation models. Existing toolkits mainly focus on fully supervised segmentation and require full and accurate pixel-level annotations that are time-consuming and difficult to acquire for segmentation tasks, which makes learning from imperfect labels highly desired for reducing the annotation cost. We aim to develop a new deep learning toolkit to support annotation-efficient learning for medical image segmentation. Methods: Our proposed toolkit named PyMIC is a modular de…
LIPN-UMR-CNRS 7030（法国国家科学研究中心7030联合研究单元LIPN实验室）；Sorbonne University Paris Nord（巴黎北索邦大学）；Amirkabir University of Technology（阿米尔卡比尔理工大学）；Laboratory of Images, Signals and Intelligent Systems（图像、信号与智能系统实验室）；Université Paris-Est Créteil（巴黎东克雷泰伊大学）
Manual segmentation of medical images (e.g., segmenting tumors in CT scans) is a high-effort task that can be accelerated with machine learning techniques. However, selecting the right segmentation approach depends on the evaluation function, particularly in medical image segmentation where we must deal with dependency between voxels. For instance, in contrast to classical systems where the predictions are either correct or incorrect, predictions in medical image segmentation may be partially correct and incorrect simultaneously. In this paper, we explore this expressiveness to extract the use…
Beihang University（北京航空航天大学）；Hefei Innovation Research Institute, Beihang University（北京航空航天大学合肥创新研究院）；Beijing Advanced Innovation Centre for Biomedical Engineering（北京生物医学工程先进创新中心）；Beijing Advanced Innovation Centre for Big Data-Based Precision Medicine（北京大数据精准医疗先进创新中心）
Recently, deep convolutional neural networks have achieved great success for medical image segmentation. However, unlike segmentation of natural images, most medical images such as MRI and CT are volumetric data. In order to make full use of volumetric information, 3D CNNs are widely used. However, 3D CNNs suffer from higher inference time and computation cost, which hinders their further clinical applications. Additionally, with the increased number of parameters, the risk of overfitting is higher, especially for medical images where data and annotations are expensive to acquire. To issue thi…
Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）
Deep convolutional neural networks (CNNs) have been widely used for medical image segmentation. In most studies, only the output layer is exploited to compute the final segmentation results and the hidden representations of the deep learned features have not been well understood. In this paper, we propose a prototype segmentation (ProtoSeg) method to compute a binary segmentation map based on deep features. We measure the segmentation abilities of the features by computing the Dice between the feature segmentation map and ground-truth, named as the segmentation ability score (SA score for shor…
Institute for Machine Learning, ETH Zurich（苏黎世联邦理工学院机器学习研究所）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Champalimaud Foundation（尚帕利莫基金会）
In clinical practice, regions of interest in medical imaging often need to be identified through a process of precise image segmentation. The quality of this image segmentation step critically affects the subsequent clinical assessment of the patient status. To enable high accuracy, automatic image segmentation, we introduce a novel deep neural network architecture that exploits the inherent spatial coherence of anatomical structures and is well equipped to capture long-range spatial dependencies in the segmented pixel/voxel space. In contrast to the state-of-the-art solutions based on convolu…
University of Science and Technology Beijing（北京科技大学）；University of Central Florida（中佛罗里达大学）
Transformer, benefiting from global (long-range) information modeling using self-attention mechanism, has been successful in natural language processing and computer vision recently. Convolutional Neural Networks, capable of capturing local features, are difficult to model explicit long-distance dependencies from global feature space. However, both local and global features are crucial for dense prediction tasks, especially for 3D medical image segmentation. In this paper, we present the further attempt to exploit Transformer in 3D CNN for 3D medical image volumetric segmentation and propose a…
Vanderbilt University（范德堡大学）；The George Washington University（乔治华盛顿大学）；Korea University（高丽大学）
Segmenting medical images accurately and reliably is important for disease diagnosis and treatment. It is a challenging task because of the wide variety of objects' sizes, shapes, and scanning modalities. Recently, many convolutional neural networks (CNN) have been designed for segmentation tasks and achieved great success. Few studies, however, have fully considered the sizes of objects, and thus most demonstrate poor performance for small objects segmentation. This can have a significant impact on the early detection of diseases. This paper proposes a Context Axial Reserve Attention Network …
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
We introduce a novel frame-interpolation-based method for slice imputation to improve segmentation accuracy for anisotropic 3D medical images, in which the number of slices and their corresponding segmentation labels can be increased between two consecutive slices in anisotropic 3D medical volumes. Unlike previous inter-slice imputation methods, which only focus on the smoothness in the axial direction, this study aims to improve the smoothness of the interpolated 3D medical volumes in all three directions: axial, sagittal, and coronal. The proposed multitask inter-slice imputation method, in …
Vanderbilt University（范德比尔特大学）；The George Washington University（乔治华盛顿大学）
Segmenting medical images accurately and reliably is important for disease diagnosis and treatment. It is a challenging task because of the wide variety of objects' sizes, shapes, and scanning modalities. Recently, many convolutional neural networks (CNN) have been designed for segmentation tasks and achieved great success. Few studies, however, have fully considered the sizes of objects, and thus most demonstrate poor performance for small objects segmentation. This can have a significant impact on the early detection of diseases. This paper proposes a Context Axial Reverse Attention Network …
From CNN to Transformer: A Review of Medical Image Segmentation Models
University of Electronic Science and Technology of China（电子科技大学）；Deyang People’s Hospital（德阳市人民医院）；Sichuan Provincial People’s Hospital（四川省人民医院）；Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital（中国科学院四川转化医学研究医院）
The Domain Shift Problem of Medical Image Segmentation and Vendor-Adaptation by Unet-GAN
Fudan University（复旦大学）；Ruijin Hospital, Shanghai Jiaotong University（上海交通大学医学院附属瑞金医院）；Tongji Hospital, Huazhong University of Science and Technology（华中科技大学同济医学院附属同济医院）；Leiden University Medical Center（莱顿大学医学中心）
UNSW（新南威尔士大学）；Dongguk University（东国大学）；Defence Science and Technology Group（国防科学技术集团）
Image segmentation is an important task in medical imaging. It constitutes the backbone of a wide variety of clinical diagnostic methods, treatments, and computer-aided surgeries. In this paper, we propose a multi-kernel image segmentation net (MKIS-Net), which uses multiple kernels to create an efficient receptive field and enhance segmentation performance. As a result of its multi-kernel design, MKIS-Net is a light-weight architecture with a small number of trainable parameters. Moreover, these multi-kernel receptive fields also contribute to better segmentation results. We demonstrate the e…
SQA-SAM: Segmentation Quality Assessment for Medical Images Utilizing the Segment Anything Model
Nanjing University of Science and Technology（南京理工大学）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学研究中心）；The Chinese University of Hong Kong（香港中文大学）；University of Notre Dame（圣母大学）
The scarcity of pixel-level annotation is a prevalent problem in medical image segmentation tasks. In this paper, we introduce a novel regularization strategy involving interpolation-based mixing for semi-supervised medical image segmentation. The proposed method is a new consistency regularization strategy that encourages segmentation of interpolation of two unlabelled data to be consistent with the interpolation of segmentation maps of those data. This method represents a specific type of data-adaptive regularization paradigm which aids to minimize the overfitting of labelled data under high…
In-context learning for medical image segmentation
Tohoku University Graduate School of Medicine（东北大学大学院医学系研究科）；Tohoku University Hospital（东北大学医院）；St. Marianna University Graduate School of Medicine（圣玛丽安娜医科大学大学院医学系研究科）
University of Georgia（佐治亚大学）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）；Shanghai Jiao Tong University（上海交通大学）
In this paper we consider the problem of unsupervised anomaly segmentation in medical images, which has attracted increasing attention in recent years due to the expensive pixel-level annotations from experts and the existence of a large amount of unannotated normal and abnormal image scans. We introduce a segmentation network that utilizes adversarial learning to partition an image into two cuts, with one of them falling into a reference distribution provided by the user. This Adversarial-based Selective Cutting network (ASC-Net) bridges the two domains of cluster-based deep segmentation and …
University of Cambridge（剑桥大学）；University of Salerno（萨莱诺大学）
Automatic segmentation methods are an important advancement in medical image analysis. Machine learning techniques, and deep neural networks in particular, are the state-of-the-art for most medical image segmentation tasks. Issues with class imbalance pose a significant challenge in medical datasets, with lesions often occupying a considerably smaller volume relative to the background. Loss functions used in the training of deep learning algorithms differ in their robustness to class imbalance, with direct consequences for model convergence. The most commonly used loss functions for segmentati…
University of Augsburg（奥格斯堡大学）；University Hospital Augsburg（奥格斯堡大学医院）
Neural networks represent a field of research that can quickly achieve very good results in the field of medical image segmentation using a GPU. A possible way to achieve good results without GPUs are random forests. For this purpose, two random forest approaches were compared with a state-of-the-art deep convolutional neural network. To make the comparison the PhC-C2DH-U373 and the retinal imaging datasets were used. The evaluation showed that the deep convolutional neutral network achieved the best results. However, one of the random forest approaches also achieved a similar high performance…
Automated segmentation of pathological regions of interest aids medical image diagnostics and follow-up care. However, accurate pathological segmentations require high quality of annotated data that can be both cost and time intensive to generate. In this work, we propose an automated two-step method that detects a minimal image subset required to train segmentation models by evaluating the quality of medical images from 3D image stacks using a U-net++ model. These images that represent a lack of quality training can then be annotated and used to fully train a U-net-based segmentation model. T…
Recent advances of semantic image segmentation greatly benefit from deeper and larger Convolutional Neural Network (CNN) models. Compared to image segmentation in the wild, properties of both medical images themselves and of existing medical datasets hinder training deeper and larger models because of overfitting. To this end, we propose a novel two-stream UNET architecture for automatic end-to-end medical image segmentation, in which intensity value and gradient vector flow (GVF) are two inputs for each stream, respectively. We demonstrate that two-stream CNNs with more low-level features gre…
Imperial College London（帝国理工学院）；BioMedIA Group, Department of Computing（计算系BioMedIA小组）
Few-shot semantic segmentation (FSS) has great potential for medical imaging applications. Most of the existing FSS techniques require abundant annotated semantic classes for training. However, these methods may not be applicable for medical images due to the lack of annotations. To address this problem we make several contributions: (1) A novel self-supervised FSS framework for medical images in order to eliminate the requirement for annotations during training. Additionally, superpixel-based pseudo-labels are generated to provide supervision; (2) An adaptive local prototype pooling module pl…
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
University of Oxford（牛津大学）；National University of Singapore（新加坡国立大学）；University of Alberta（阿尔伯塔大学）；A*STAR（新加坡科技研究局）；Carnegie Mellon University（卡内基梅隆大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Hongkong（香港大学）
本文提出Medical SAM Adapter（Med-SA），通过轻量适配技术将医学知识融入SAM，利用SD-Trans和HyP-Adpt适配2D模型至3D医学图像，在17个任务上以仅2%参数更新超越SOTA方法。
Université de technologie de Compiègne（贡比涅技术大学）；University of Rouen Normandy（诺曼底鲁昂大学）；Institut universitaire de France（法国大学研究院）
The investigation of uncertainty is of major importance in risk-critical applications, such as medical image segmentation. Belief function theory, a formal framework for uncertainty analysis and multiple evidence fusion, has made significant contributions to medical image segmentation, especially since the development of deep learning. In this paper, we provide an introduction to the topic of medical image segmentation methods using belief function theory. We classify the methods according to the fusion step and explain how information with uncertainty or imprecision is modeled and fused with …
SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation
School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息工程学院）；Pazhou Lab（琶洲实验室）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）
Illinois Institute of Technology（伊利诺伊理工学院）；University of Michigan（密歇根大学）
Recently, due to the increasing requirements of medical imaging applications and the professional requirements of annotating medical images, few-shot learning has gained increasing attention in the medical image semantic segmentation field. To perform segmentation with limited number of labeled medical images, most existing studies use Proto-typical Networks (PN) and have obtained compelling success. However, these approaches overlook the query image features extracted from the proposed representation network, failing to preserving the spatial connection between query and support images. In th…
Northwestern Polytechnical University（西北工业大学）；The University of Adelaide（阿德莱德大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）
Manual annotation of medical images is highly subjective, leading to inevitable and huge annotation biases. Deep learning models may surpass human performance on a variety of tasks, but they may also mimic or amplify these biases. Although we can have multiple annotators and fuse their annotations to reduce stochastic errors, we cannot use this strategy to handle the bias caused by annotators' preferences. In this paper, we highlight the issue of annotator-related biases on medical image segmentation tasks, and propose a Preference-involved Annotation Distribution Learning (PADL) framework to …
