We present Token-UNet, adopting the TokenLearner and TokenFuser modules to encase Transformers into UNets.
While Transformers have enabled global interactions among input elements in medical imaging, current computational challenges hinder their deployment on common hardware. Models like (Swin)UNETR adapt the UNet architecture by incorporating (Swin)Transformer encoders, which process tokens that each represent small subvolumes ($8^3$ voxels) of the input.
The Transformer attention mechanism scales quadratically with the number of tokens, which is tied to the cubic scaling of 3D input resolution.
This work reconsiders the role of convolution and attention, introducing Token-UNets, a family of 3D segmentation models that can operate in constrained computational environments and time frames.
To mitigate computational demands, our approach maintains the convolutional encoder of UNet-like models, and applies TokenLearner to 3D feature maps. This module pools a preset number of tokens from local and global structures.
Our results show this tokenization effectively encodes task-relevant information, yielding naturally interpretable attention maps. The memory footprint, computation times at inference, and parameter counts of our heaviest model are reduced to 33\%, 10\%, and 35\% of the SwinUNETR values, with better average performance (86.75\% $\pm 0.19\%$ Dice score for SwinUNETR vs our 87.21\% $\pm 0.35\%$).
This work opens the way to more efficient trainings in contexts with limited computational resources, such as 3D medical imaging. Easing model optimization, fine-tuning, and transfer-learning in limited hardware settings can accelerate and diversify the development of approaches, for the benefit of the research community.
Huazhong University of Science and Technology（华中科技大学）；Shenzhen University（深圳大学）
Deep convolutional neural network (CNN) achieves remarkable performance for medical image analysis. UNet is the primary source in the performance of 3D CNN architectures for medical imaging tasks, including brain tumor segmentation. The skip connection in the UNet architecture concatenates features from both encoder and decoder paths to extract multi-contextual information from image data. The multi-scaled features play an essential role in brain tumor segmentation. However, the limited use of features can degrade the performance of the UNet approach for segmentation. In this paper, we propose…
Squeeze Excitation Embedded Attention UNet for Brain Tumor Segmentation
School of Electronics Engineering, Vellore Institute of Technology（韦洛尔理工大学电子工程学院）；Centre for Advanced Data Science, Vellore Institute of Technology（韦洛尔理工大学高级数据科学中心）
Ashford and St Peter’s Hospitals NHS Foundation Trust（阿什福德和圣彼得医院 NHS 基金会信托）；University of Lincoln（林肯大学）
In this paper we propose a 2D deep residual Unet with 104 convolutional layers (DR-Unet104) for lesion segmentation in brain MRIs. We make multiple additions to the Unet architecture, including adding the 'bottleneck' residual block to the Unet encoder and adding dropout after each convolution block stack. We verified the effect of introducing the regularisation of dropout with small rate (e.g. 0.2) on the architecture, and found a dropout of 0.2 improved the overall performance compared to no dropout, or a dropout of 0.5. We evaluated the proposed architecture as part of the Multimodal Brain …
Convolutional neural networks (CNNs) have achieved remarkable success in automatically segmenting organs or lesions on 3D medical images. Recently, vision transformer networks have exhibited exceptional performance in 2D image classification tasks. Compared with CNNs, transformer networks have an appealing advantage of extracting long-range features due to their self-attention algorithm. Therefore, we propose a CNN-Transformer combined model, called BiTr-Unet, with specific modifications for brain tumor segmentation on multi-modal MRI scans. Our BiTr-Unet achieves good performance on the BraTS…
Automatic extraction of liver and tumor from CT volumes is a challenging task due to their heterogeneous and diffusive shapes. Recently, 2D and 3D deep convolutional neural networks have become popular in medical image segmentation tasks because of the utilization of large labeled datasets to learn hierarchical features. However, 3D networks have some drawbacks due to their high cost on computational resources. In this paper, we propose a 3D hybrid residual attention-aware segmentation method, named RA-UNet, to precisely extract the liver volume of interests (VOI) and segment tumors from the l…
Faculty of Computer Science, AGH University of Krakow（计算机科学系，克拉科夫AGH大学）；Department of Informatics, Universitas Pembangunan Nasional Veteran Yogyakarta（信息系，全国 veterans 大学 Yogya 市）
Universitat Politècnica de Catalunya. BarcelonaTech（加泰罗尼亚理工大学·巴塞罗那理工）
Automation of brain tumor segmentation in 3D magnetic resonance images (MRIs) is key to assess the diagnostic and treatment of the disease. In recent years, convolutional neural networks (CNNs) have shown improved results in the task. However, high memory consumption is still a problem in 3D-CNNs. Moreover, most methods do not include uncertainty information, which is especially critical in medical diagnosis. This work studies 3D encoder-decoder architectures trained with patch-based techniques to reduce memory consumption and decrease the effect of unbalanced data. The different trained model…
Hybrid Multihead Attentive Unet-3D for Brain Tumor Segmentation
University of Engineering and Technology, Lahore（拉合尔工程技术大学）；School of Interdisciplinary Engineering and Sciences, National University of Sciences and Technology（国家科技大学跨学科工程与科学学院）
New Jersey Institute of Technology（新泽西理工学院）；Kessler Foundation（凯斯勒基金会）；Rutgers – New Jersey Medical School（罗格斯大学新泽西医学院）；Medical College of Wisconsin（威斯康星医学院）；Rutgers University – Newark（罗格斯大学纽瓦克分校）
Automatic identification of brain lesions from magnetic resonance imaging (MRI) scans of stroke survivors would be a useful aid in patient diagnosis and treatment planning. We propose a multi-modal multi-path convolutional neural network system for automating stroke lesion segmentation. Our system has nine end-to-end UNets that take as input 2-dimensional (2D) slices and examines all three planes with three different normalizations. Outputs from these nine total paths are concatenated into a 3D volume that is then passed to a 3D convolutional neural network to output a final lesion mask. We tr…
MAProtoNet: A Multi-scale Attentive Interpretable Prototypical Part Network for 3D Magnetic Resonance Imaging Brain Tumor Classification
Tokyo University of Agriculture and Technology (TUAT)（东京农工大学）；RIKEN Center for Advanced Intelligence Project (RIKEN AIP)（理化学研究所先进智能研究中心）；Graduate School of Medicine, Juntendo University（顺天堂大学医学院）；Faculty of Health Data Science, Juntendo University（顺天堂大学健康数据科学学部）；RIKEN（理化学研究所）
Northeastern University（东北大学）；Shengjing Hospital, China Medical University（中国医科大学附属盛京医院）
In this paper, we propose a dual aggregation network to adaptively aggregate different information in infant brain MRI segmentation. More precisely, we added two modules based on 3D-UNet to better model information at different levels and locations. The dilated convolution pyramid downsampling module is mainly to solve the problem of loss of spatial information on the downsampling process, and it can effectively save details while reducing the resolution. The self-attention module can integrate the remote dependence on the feature maps in two dimensions of spatial and channel, effectively impr…
SST-DUNet: Automated preclinical functional MRI skull stripping using Smart Swin Transformer and Dense UNet
School of Information Technology, Carleton University（卡尔顿大学信息技术学院）；Center for Translational NeuroImaging (CTNI), Northeastern University（东北大学转化神经影像学中心）；College of Science and Humanity Studies, Prince Sattam Bin Abdulaziz University（萨塔姆·本·阿卜杜勒阿齐兹王子大学科学与人文研究学院）；Tessellis Ltd.（泰塞利斯有限公司）
An Intuitionistic Fuzzy Logic Driven UNet architecture: Application to Brain Image segmentation
Department of Mathematics, Bareilly College, Bareilly (MJP Rohilkhand University), Uttar Pradesh, India（印度乌塔兰普尔邦巴雷利学院数学系）；Division of Newborn Medicine, Fetal Neonatal Neuroimaging and Developmental Science Center, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, USA（美国哈佛医学院波士顿儿童医院新生儿医学部）；Department of Pediatrics, Harvard Medical School, Boston, MA, USA（美国哈佛医学院儿科系）；CSIR–Central Electronics Engineering Research Institute, Pilani 333031, Rajasthan, India（印度拉贾斯坦邦比拉尼中央电子工程研究机构）；Academy of Scientific and Innovative Research (AcSIR), Ghaziabad 201002, India（印度德里阿齐亚德瓦学术与创新研究院）；CSIR–National Institute of Science Communication and Policy Research, New Delhi, India（印度新德里国家科学传播与政策研究所）
An Ensemble Approach for Brain Tumor Segmentation and Synthesis
Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗G.艾伦计算机科学学院）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Department of Bioengineering, University of Washington（华盛顿大学生物工程系）；Department of Electrical and Computer Engineering, University of Washington（华盛顿大学电气与计算机工程系）；Department of Mechanical Engineering, University of Washington（华盛顿大学机械工程系）；Watauga High School（沃托加高中）；Department of Neurological Surgery, University of Washington（华盛顿大学神经外科系）
Demystifying Deep Learning-based Brain Tumor Segmentation with 3D UNets and Explainable AI (XAI): A Comparative Analysis
School of Artificial Intelligence and Robotics, Xiamen University Malaysia（人工智能与机器人学院，马来西亚厦门大学）；School of Computing and Data Science, Xiamen University Malaysia（计算与数据科学学院，马来西亚厦门大学）；Georgia Institute of Technology（佐治亚理工学院）；School of Social and Health Sciences, James Cook University（社会科学与健康科学学院，詹姆斯库克大学）；College of Healthcare Sciences, James Cook University（健康科学学院，詹姆斯库克大学）
Samsung Medical Center（三星医疗中心）；Sungkyunkwan University School of Medicine（成均馆大学医学院）；Sungkyunkwan University（成均馆大学）；Columbia University（哥伦比亚大学）；University of Maryland（马里兰大学）
Automated segmentation and volumetry of brain magnetic resonance imaging (MRI) scans are essential for the diagnosis of Parkinson's disease (PD) and Parkinson's plus syndromes (P-plus). To enhance the diagnostic performance, we adopt deep learning (DL) models in brain segmentation and compared their performance with the gold-standard non-DL method. We collected brain MRI scans of healthy controls (n=105) and patients with PD (n=105), multiple systemic atrophy (n=132), and progressive supranuclear palsy (n=69) at Samsung Medical Center from January 2017 to December 2020. Using the gold-standard…
Interpretable Motion Artificat Detection in structural Brain MRI
Indian Institute of Technology, Madras（印度理工学院，马德拉斯）；Boston Childrens Hospital, Harvard Medical School（波士顿儿童医院，哈佛医学院）；Indian Institute of Technology, Bhilai（印度理工学院，比哈尔）
ResNet-34 with Lightweight Decoder for Accurate and Efficient Segmentation of Fetal Brain MRI
Department of Biomedical Physics and Technology, University of Dhaka（达卡大学生物医学物理与技术系）；Department of Electrical Engineering, College of Engineering, Qatar University（卡塔尔大学工程学院电气工程系）；Department of Biomedical Engineering, Jashore University of Science and Technology（贾沙尔大学科学与技术学院生物医学工程系）
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Brain tumor segmentation is essential for the diagnosis and prognosis of patients with gliomas. The brain tumor segmentation challenge has continued to provide a great source of data to develop automatic algorithms to perform the task. This paper describes our contribution to the 2021 competition. We developed our methods based on nn-UNet, the winning entry of last year competition. We experimented with several modifications, including using a larger network, replacing batch normalization with group normalization, and utilizing axial attention in the decoder. Internal 5-fold cross validation a…
Shenzhen University（深圳大学）；Huazhong University of Science and Technology（华中科技大学）
The brain tumor segmentation task aims to classify tissue into the whole tumor (WT), tumor core (TC), and enhancing tumor (ET) classes using multimodel MRI images. Quantitative analysis of brain tumors is critical for clinical decision making. While manual segmentation is tedious, time-consuming, and subjective, this task is at the same time very challenging to automatic segmentation methods. Thanks to the powerful learning ability, convolutional neural networks (CNNs), mainly fully convolutional networks, have shown promising brain tumor segmentation. This paper further boosts the performance…
NUS Graduate School for Integrative Sciences and Engineering, NUS（新加坡国立大学综合科学与工程研究生院）；National University of Singapore（新加坡国立大学）；NIT, Tiruchirappalli（蒂鲁吉拉伯利国立理工学院）；University of Moratuwa（莫拉图瓦大学）；Punjab Engineering College（旁遮普工程学院）
In this work, we develop an attention convolutional neural network (CNN) to segment brain tumors from Magnetic Resonance Images (MRI). Further, we predict the survival rate using various machine learning methods. We adopt a 3D UNet architecture and integrate channel and spatial attention with the decoder network to perform segmentation. For survival prediction, we extract some novel radiomic features based on geometry, location, the shape of the segmented tumor and combine them with clinical information to estimate the survival duration for each patient. We also perform extensive experiments t…
Brain Tumor Segmentation in MRI Images with 3D U-Net and Contextual Transformer
University of Information Technology（信息科技大学）；Vietnam National University（越南国家大学）；College of Technology and Design, University of Economics Ho Chi Minh City（胡志明市经济大学技术与设计学院）；Ho Chi Minh City Open University（胡志明市开放大学）
Vision Transformers (ViT)s have recently become popular due to their outstanding modeling capabilities, in particular for capturing long-range information, and scalability to dataset and model sizes which has led to state-of-the-art performance in various computer vision and medical image analysis tasks. In this work, we introduce a unified framework consisting of two architectures, dubbed UNetFormer, with a 3D Swin Transformer-based encoder and Convolutional Neural Network (CNN) and transformer-based decoders. In the proposed model, the encoder is linked to the decoder via skip connections at…
Otto von Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）；Human Technopole（人类技术园）；German Centre for Neurodegenerative Disease（德国神经退行性疾病中心）；Centre for Behavioural Brain Sciences（行为脑科学中心）
Computed tomography and magnetic resonance imaging are two widely used clinical imaging modalities for non-invasive diagnosis. However, both of these modalities come with certain problems. CT uses harmful ionising radiation, and MRI suffers from slow acquisition speed. Both problems can be tackled by undersampling, such as sparse sampling. However, such undersampled data leads to lower resolution and introduces artefacts. Several techniques, including deep learning based methods, have been proposed to reconstruct such data. However, the undersampled reconstruction problem for these two modalit…
Lobachevsky State University（罗巴切夫斯基国立大学）；Intel（英特尔）
MRI analysis takes central position in brain tumor diagnosis and treatment, thus it's precise evaluation is crucially important. However, it's 3D nature imposes several challenges, so the analysis is often performed on 2D projections that reduces the complexity, but increases bias. On the other hand, time consuming 3D evaluation, like, segmentation, is able to provide precise estimation of a number of valuable spatial characteristics, giving us understanding about the course of the disease.\newline Recent studies, focusing on the segmentation task, report superior performance of Deep Learning …
Gliomas are the most common and aggressive among brain tumors, which cause a short life expectancy in their highest grade. Therefore, treatment assessment is a key stage to enhance the quality of the patients' lives. Recently, deep convolutional neural networks (DCNNs) have achieved a remarkable performance in brain tumor segmentation, but this task is still difficult owing to high varying intensity and appearance of gliomas. Most of the existing methods, especially UNet-based networks, integrate low-level and high-level features in a naive way, which may result in confusion for the model. Mor…
Semantic segmentation of brain tumors is a fundamental medical image analysis task involving multiple MRI imaging modalities that can assist clinicians in diagnosing the patient and successively studying the progression of the malignant entity. In recent years, Fully Convolutional Neural Networks (FCNNs) approaches have become the de facto standard for 3D medical image segmentation. The popular "U-shaped" network architecture has achieved state-of-the-art performance benchmarks on different 2D and 3D semantic segmentation tasks and across various imaging modalities. However, due to the limited…
Sorbonne Université（索邦大学）；Institut du Cerveau - Paris Brain Institute - ICM（巴黎大脑研究所（ICM））；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共援助医院集团）；Hôpital de la Pitié Salpêtrière（皮蒂耶-萨尔佩特里埃医院）
Early and accurate diagnosis of parkinsonian syndromes is critical to provide appropriate care to patients and for inclusion in therapeutic trials. The red nucleus is a structure of the midbrain that plays an important role in these disorders. It can be visualized using iron-sensitive magnetic resonance imaging (MRI) sequences. Different iron-sensitive contrasts can be produced with MRI. Combining such multimodal data has the potential to improve segmentation of the red nucleus. Current multimodal segmentation algorithms are computationally consuming, cannot deal with missing modalities and ne…
