Medical image analysis using deep learning has recently been prevalent, showing great performance for various downstream tasks including medical image segmentation and its sibling, volumetric image segmentation. Particularly, a typical volumetric segmentation network strongly relies on a voxel grid representation which treats volumetric data as a stack of individual voxel `slices', which allows learning to segment a voxel grid to be as straightforward as extending existing image-based segmentation networks to the 3D domain. However, using a voxel grid representation requires a large memory footprint, expensive test-time and limiting the scalability of the solutions. In this paper, we propose Point-Unet, a novel method that incorporates the efficiency of deep learning with 3D point clouds into volumetric segmentation. Our key idea is to first predict the regions of interest in the volume by learning an attentional probability map, which is then used for sampling the volume into a sparse point cloud that is subsequently segmented using a point-based neural network. We have conducted the experiments on the medical volumetric segmentation task with both a small-scale dataset Pancreas and large-scale datasets BraTS18, BraTS19, and BraTS20 challenges. A comprehensive benchmark on different metrics has shown that our context-aware Point-Unet robustly outperforms the SOTA voxel-based networks at both accuracies, memory usage during training, and time consumption during testing. Our code is available at https://github.com/VinAIResearch/Point-Unet.
Unets have become the standard method for semantic segmentation of medical images, along with fully convolutional networks (FCN). Unet++ was introduced as a variant of Unet, in order to solve some of the problems facing Unet and FCNs. Unet++ provided networks with an ensemble of variable depth Unets, hence eliminating the need for professionals estimating the best suitable depth for a task. While Unet and all its variants, including Unet++ aimed at providing networks that were able to train well without requiring large quantities of annotated data, none of them attempted to eliminate the need …
UNetMamba: An Efficient UNet-Like Mamba for Semantic Segmentation of High-Resolution Remote Sensing Images
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Key Laboratory of Target Cognition and Application Technology, CAS（中国科学院目标认知与应用技术重点实验室）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Recent advances of semantic image segmentation greatly benefit from deeper and larger Convolutional Neural Network (CNN) models. Compared to image segmentation in the wild, properties of both medical images themselves and of existing medical datasets hinder training deeper and larger models because of overfitting. To this end, we propose a novel two-stream UNET architecture for automatic end-to-end medical image segmentation, in which intensity value and gradient vector flow (GVF) are two inputs for each stream, respectively. We demonstrate that two-stream CNNs with more low-level features gre…
CM-UNet: Hybrid CNN-Mamba UNet for Remote Sensing Image Semantic Segmentation
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；School of Aeronautics and Astronautics, Zhejiang University（浙江大学航空航天学院）；College of Computer Science, Zhejiang University（浙江大学计算机学院）
DSVM-UNet : Enhancing VM-UNet with Dual Self-distillation for Medical Image Segmentation
Faculty of Military Health Service, Naval Medical University, China（海军医学院军事卫生服务学院）；School of Computer Science and Technology, Shanghai University of Electric Power, China（上海电力大学计算机科学与技术学院）
Wuhan Institute of Technology（武汉工程大学）；Huazhong University of Science and technology（华中科技大学）
Medical image segmentation plays an essential role in developing computer-assisted diagnosis and therapy systems, yet still faces many challenges. In the past few years, the popular encoder-decoder architectures based on CNNs (e.g., U-Net) have been successfully applied in the task of medical image segmentation. However, due to the locality of convolution operations, they demonstrate limitations in learning global context and long-range spatial relations. Recently, several researchers try to introduce transformers to both the encoder and decoder components with promising results, but the effic…
National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）
This paper reports Deep LOGISMOS approach to 3D tumor segmentation by incorporating boundary information derived from deep contextual learning to LOGISMOS - layered optimal graph image segmentation of multiple objects and surfaces. Accurate and reliable tumor segmentation is essential to tumor growth analysis and treatment selection. A fully convolutional network (FCN), UNet, is first trained using three adjacent 2D patches centered at the tumor, providing contextual UNet segmentation and probability map for each 2D patch. The UNet segmentation is then refined by Gaussian Mixture Model (GMM) a…
An Intuitionistic Fuzzy Logic Driven UNet architecture: Application to Brain Image segmentation
Department of Mathematics, Bareilly College, Bareilly (MJP Rohilkhand University), Uttar Pradesh, India（印度乌塔兰普尔邦巴雷利学院数学系）；Division of Newborn Medicine, Fetal Neonatal Neuroimaging and Developmental Science Center, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, USA（美国哈佛医学院波士顿儿童医院新生儿医学部）；Department of Pediatrics, Harvard Medical School, Boston, MA, USA（美国哈佛医学院儿科系）；CSIR–Central Electronics Engineering Research Institute, Pilani 333031, Rajasthan, India（印度拉贾斯坦邦比拉尼中央电子工程研究机构）；Academy of Scientific and Innovative Research (AcSIR), Ghaziabad 201002, India（印度德里阿齐亚德瓦学术与创新研究院）；CSIR–National Institute of Science Communication and Policy Research, New Delhi, India（印度新德里国家科学传播与政策研究所）
VinAI Research（VinAI研究院）；University of Arkansas（阿肯色大学）
Medical image segmentation has been so far achieving promising results with Convolutional Neural Networks (CNNs). However, it is arguable that in traditional CNNs, its pooling layer tends to discard important information such as positions. Moreover, CNNs are sensitive to rotation and affine transformation. Capsule network is a data-efficient network design proposed to overcome such limitations by replacing pooling layers with dynamic routing and convolutional strides, which aims to preserve the part-whole relationships. Capsule network has shown a great performance in image recognition and nat…
Wuhan University（武汉大学）；Key Laboratory of Natural Resources Monitoring in Tropical and Subtropical Area of South China, Ministry of Natural Resources（自然资源部华南热带亚热带自然资源监测重点实验室）；Lancaster University（兰卡斯特大学）；UK Centre for Ecology & Hydrology（英国生态与水文研究中心）；University of Twente（特文特大学）；University of Southampton（南安普顿大学）；Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences（中国科学院地理科学与资源研究所）
Semantic segmentation of remotely sensed urban scene images is required in a wide range of practical applications, such as land cover mapping, urban change detection, environmental protection, and economic assessment.Driven by rapid developments in deep learning technologies, the convolutional neural network (CNN) has dominated semantic segmentation for many years. CNN adopts hierarchical feature representation, demonstrating strong capabilities for local information extraction. However, the local property of the convolution layer limits the network from capturing the global context. Recently,…
SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Computer and Information Engineering, Jiangxi Normal University（江西师范大学计算机与信息工程学院）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Tianjin University（天津大学）；Monash University（墨尔本大学）；Pazhou Lab（琶洲实验室）；Smart Hospital Research Institute, Peking University Shenzhen Hospital（北京大学深圳医院智能医院研究所）
Optimizing Neural Network Architecture for Medical Image Segmentation Using Monte Carlo Tree Search
Xi'an Kedagaoxin University, Faculty of Science（西安科大新大学科学学院）；Xi'an Kedagaoxin University, School of International Education（西安科大新大学国际教育学院）；Saint Petersburg State University（圣彼得堡国立大学）
Lobachevsky State University（罗巴切夫斯基国立大学）；Intel（英特尔）
MRI analysis takes central position in brain tumor diagnosis and treatment, thus it's precise evaluation is crucially important. However, it's 3D nature imposes several challenges, so the analysis is often performed on 2D projections that reduces the complexity, but increases bias. On the other hand, time consuming 3D evaluation, like, segmentation, is able to provide precise estimation of a number of valuable spatial characteristics, giving us understanding about the course of the disease.\newline Recent studies, focusing on the segmentation task, report superior performance of Deep Learning …
TM-UNet: Token-Memory Enhanced Sequential Modeling for Efficient Medical Image Segmentation
Dalian University of Technology（大连理工大学）；University of Lincoln（林肯大学）；University of Nottingham（诺丁汉大学）；Waseda University（早稻田大学）；University of Nottingham Ningbo China（宁波诺丁汉大学）；Yale University（耶鲁大学）
As an essential prerequisite for developing a medical intelligent assistant system, medical image segmentation has received extensive research and concentration from the neural network community. A series of UNet-like networks with encoder-decoder architecture has achieved extraordinary success, in which UNet2+ and UNet3+ redesign skip connections, respectively proposing dense skip connection and full-scale skip connection and dramatically improving compared with UNet in medical image segmentation. However, UNet2+ lacks sufficient information explored from the full scale, which will affect the…
In the isointense stage, the accurate volumetric image segmentation is a challenging task due to the low contrast between tissues. In this paper, we propose a novel very deep network architecture based on a densely convolutional network for volumetric brain segmentation. The proposed network architecture provides a dense connection between layers that aims to improve the information flow in the network. By concatenating features map of fine and coarse dense blocks, it allows capturing multi-scale contextual information. Experimental results demonstrate significant advantages of the proposed me…
Recently deep residual learning with residual units for training very deep neural networks advanced the state-of-the-art performance on 2D image recognition tasks, e.g., object detection and segmentation. However, how to fully leverage contextual representations for recognition tasks from volumetric data has not been well studied, especially in the field of medical image computing, where a majority of image modalities are in volumetric format. In this paper we explore the deep residual learning on the task of volumetric brain segmentation. There are at least two main contributions in our work.…
Although convolutional neural networks (CNNs) are promoting the development of medical image semantic segmentation, the standard model still has some shortcomings. First, the feature mapping from the encoder and decoder sub-networks in the skip connection operation has a large semantic difference. Second, the remote feature dependence is not effectively modeled. Third, the global context information of different scales is ignored. In this paper, we try to eliminate semantic ambiguity in skip connection operations by adding attention gates (AGs), and use attention mechanisms to combine local fe…
The Chinese University of Hong Kong（香港中文大学）；Imsight Medical Technology, Inc.（伊赛智能医疗科技有限公司）
Liver cancer is one of the leading causes of cancer death. To assist doctors in hepatocellular carcinoma diagnosis and treatment planning, an accurate and automatic liver and tumor segmentation method is highly demanded in the clinical practice. Recently, fully convolutional neural networks (FCNs), including 2D and 3D FCNs, serve as the back-bone in many volumetric image segmentation. However, 2D convolutions can not fully leverage the spatial information along the third dimension while 3D convolutions suffer from high computational cost and GPU memory consumption. To address these issues, we …
In 3D medical image segmentation, small targets segmentation is crucial for diagnosis but still faces challenges. In this paper, we propose the Axis Projection Attention UNet, named APAUNet, for 3D medical image segmentation, especially for small targets. Considering the large proportion of the background in the 3D feature space, we introduce a projection strategy to project the 3D features into three orthogonal 2D planes to capture the contextual attention from different views. In this way, we can filter out the redundant feature information and mitigate the loss of critical information for s…
We propose an attention mechanism for 3D medical image segmentation. The method, named segmentation-by-detection, is a cascade of a detection module followed by a segmentation module. The detection module enables a region of interest to come to attention and produces a set of object region candidates which are further used as an attention model. Rather than dealing with the entire volume, the segmentation module distills the information from the potential region. This scheme is an efficient solution for volumetric data as it reduces the influence of the surrounding noise which is especially im…
SelfReg-UNet: Self-Regularized UNet for Medical Image Segmentation
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；McKeley School of Engineering, Washington University in St. Louis（圣路易斯华盛顿大学麦凯莱工程学院）；School of Computing, Clemson University（克莱姆森大学计算学院）
Beijing Zoezen Robot Co Ltd（北京佐伊森机器人有限公司）；Beihang University（北京航空航天大学）
Medical image segmentation have drawn massive attention as it is important in biomedical image analysis. Good segmentation results can assist doctors with their judgement and further improve patients' experience. Among many available pipelines in medical image analysis, Unet is one of the most popular neural networks as it keeps raw features by adding concatenation between encoder and decoder, which makes it still widely used in industrial field. In the mean time, as a popular model which dominates natural language process tasks, transformer is now introduced to computer vision tasks and have …
ACM-UNet: Adaptive Integration of CNNs and Mamba for Efficient Medical Image Segmentation
Wuhan University of Technology（武汉理工大学）；Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）
PBE-UNet: A light weight Progressive Boundary-Enhanced U-Net with Scale-Aware Aggregation for Ultrasound Image Segmentation
Department of Computer Science, Shaoxing University（绍兴大学计算机科学系）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Northeastern University（工业智能化与系统优化国家级前沿科学中心，东北大学）；College of Computer Science, Chongqing University（重庆大学计算机学院）；School of Computer and Computing Science, Hangzhou City University（杭州市大学计算机与计算科学学院）；ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC研究院）；Information Technology R&D Innovation Center of Peking University（北京大学信息技术研发创新中心）
VM-UNet: Vision Mamba UNet for Medical Image Segmentation
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）
Xiamen University（厦门大学）；The University of Hong Kong（香港大学）
Recent medical image segmentation models are mostly hybrid, which integrate self-attention and convolution layers into the non-isomorphic architecture. However, one potential drawback of these approaches is that they failed to provide an intuitive explanation of why this hybrid combination manner is beneficial, making it difficult for subsequent work to make improvements on top of them. To address this issue, we first analyze the differences between the weight allocation mechanisms of the self-attention and convolution. Based on this analysis, we propose to construct a parallel non-isomorphic …
LightM-UNet: Mamba Assists in Lightweight UNet for Medical Image Segmentation
School of Computer Science, Peking University（北京大学计算机学院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Key Laboratory of High Confidence Software Technologies, Ministry of Education（教育部高可信软件技术重点实验室）；National Engineering Research Center for Software Engineering, Peking University（北京大学国家软件工程研究中心）
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Sir Run Run Shaw Hospital（邵逸夫医院）；College of Information Science and Engineering, Ritsumeikan University（立命馆大学信息科学与工程学院）；Research Center for Healthcare Data Science, Zhejiang Lab（之江实验室医疗数据科学研究中心）
Recently, a growing interest has been seen in deep learning-based semantic segmentation. UNet, which is one of deep learning networks with an encoder-decoder architecture, is widely used in medical image segmentation. Combining multi-scale features is one of important factors for accurate segmentation. UNet++ was developed as a modified Unet by designing an architecture with nested and dense skip connections. However, it does not explore sufficient information from full scales and there is still a large room for improvement. In this paper, we propose a novel UNet 3+, which takes advantage of f…
Technische Universität München（慕尼黑工业大学）；Fudan University（复旦大学）；Huawei Technologies（华为技术有限公司）
In the past few years, convolutional neural networks (CNNs) have achieved milestones in medical image analysis. Especially, the deep neural networks based on U-shaped architecture and skip-connections have been widely applied in a variety of medical image tasks. However, although CNN has achieved excellent performance, it cannot learn global and long-range semantic information interaction well due to the locality of the convolution operation. In this paper, we propose Swin-Unet, which is an Unet-like pure Transformer for medical image segmentation. The tokenized image patches are fed into the …
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）；Seattle University（西雅图大学）
Recently, many attempts have been made to construct a transformer base U-shaped architecture, and new methods have been proposed that outperformed CNN-based rivals. However, serious problems such as blockiness and cropped edges in predicted masks remain because of transformers' patch partitioning operations. In this work, we propose a new U-shaped architecture for medical image segmentation with the help of the newly introduced focal modulation mechanism. The proposed architecture has asymmetric depths for the encoder and decoder. Due to the ability of the focal module to aggregate local and g…
It is a challenge to segment the location and size of rectal cancer tumours through deep learning. In this paper, in order to improve the ability of extracting suffi-cient feature information in rectal tumour segmentation, attention enlarged ConvNeXt UNet (AACN-UNet), is proposed. The network mainly includes two improvements: 1) the encoder stage of UNet is changed to ConvNeXt structure for encoding operation, which can not only integrate multi-scale semantic information on a large scale, but al-so reduce information loss and extract more feature information from CT images; 2) CBAM attention m…
Medical Image Segmentation Using Advanced Unet: VMSE-Unet and VM-Unet CBAM+
School of Computer Engineering, Kalinga Institute of Industrial Technology (KIIT) Deemed to be University（计算机工程学院，基灵加工业技术学院（KIIT）被认定为大学）；Department of Electrical and Electronics Engineering, BITS-Pilani, Pilani Campus（电气与电子工程系，BITS-帕利尼，帕利尼校区）
Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation
Helmholtz Pioneer Campus, Helmholtz Munich, Neuherberg, Germany（海德堡先锋校园，海德堡穆恩奇，纽赫尔伯格，德国）；School of Computation, Information and Technology, TUM, Garching, Germany（计算、信息与技术学院，技术大学慕尼黑，冈辛，德国）；Department of Chemistry, TUM, Garching, Germany（化学系，技术大学慕尼黑，冈辛，德国）
本文提出 ViTC-UNet，通过可学习令牌和双向注意力解码器将预训练 ViT 表示条件化于 UNet，以提升生物医学语义分割的精度与适应性。
VM-UNET-V2 Rethinking Vision Mamba UNet for Medical Image Segmentation
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Huazhong University of Science and Technology（华中科技大学）；Jiangsu Province Hospital of Chinese Medicine（江苏省中医院）
The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations: (1) their optimal depth is apriori unknown, requiring extensive architecture search or inefficient ensemble of models of varying depths; and (2) their skip connections impose an unnecessarily restrictive fusion scheme, forcing aggregation only at the same-scale feature maps of the encoder and decoder sub-networks. To overcome these two limitations, we propose UNet++, a new neural architecture for semantic and instance…
Recent advances in transformer-based models have drawn attention to exploring these techniques in medical image segmentation, especially in conjunction with the U-Net model (or its variants), which has shown great success in medical image segmentation, under both 2D and 3D settings. Current 2D based methods either directly replace convolutional layers with pure transformers or consider a transformer as an additional intermediate encoder between the encoder and decoder of U-Net. However, these approaches only consider the attention encoding within one single slice and do not utilize the axial-a…
