MRSegmentator: Multi-Modality Segmentation of 40 Classes in MRI and CT
Department of Radiology, Charité - Universitätsmedizin Berlin（柏林查理医院放射科）；Freie Universität Berlin（柏林自由大学）；Humboldt Universität zu Berlin（柏林洪堡大学）；Diagnostic Image Analysis Group, Radboud University Medical Center（拉德堡德大学医学中心诊断图像分析组）；School of Medicine and Health, Klinikum rechts der Isar, TUM University Hospital（医学与健康学院，莱比锡大学医院）；Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg（埃朗根大学医院放射科，埃朗根-纽伦堡弗里德里希-亚历山大大学）；Laboratory for Computer Aided Medical Procedures, Technical University of Munich（慕尼黑技术大学计算机辅助医疗程序实验室）；Chair for AI in Medicine and Healthcare, Klinikum rechts der Isar, Technical University Munich（医学与健康中的AI教授职位，莱比锡大学医院，慕尼黑技术大学）
Hartmut Häntze, Lina Xu, Christian J. Mertens, Felix J. Dorfner, Leonhard Donle, Felix Busch, Avan Kader, Sebastian Ziegelmayer, Nadine Bayerl, Nassir Navab, Da… 展开作者
Hartmut Häntze, Lina Xu, Christian J. Mertens, Felix J. Dorfner, Leonhard Donle, Felix Busch, Avan Kader, Sebastian Ziegelmayer, Nadine Bayerl, Nassir Navab, Daniel Rueckert, Julia Schnabel, Hugo JWL Aerts, Daniel Truhn, Fabian Bamberg, Jakob Weiß, Christopher L. Schlett, Steffen Ringhof, Thoralf Niendorf, Tobias Pischon, Hans-Ulrich Kauczor, Tobias Nonnenmacher, Thomas Kröncke, Henry Völzke, Jeanette Schulz-Menger, Klaus Maier-Hein, Mathias Prokop, Bram van Ginneken, Alessa Hering, Marcus R. Makowski, Lisa C. Adams, Keno K. Bressem
备注：
17 pages, 6 figures; updated data; completed co-author info
Purpose: To develop and evaluate a deep learning model for multi-organ segmentation of MRI scans.
Materials and Methods: The model was trained on 1,200 manually annotated 3D axial MRI scans from the UK Biobank, 221 in-house MRI scans, and 1228 CT scans from the TotalSegmentator dataset. A human-in-the-loop annotation workflow was employed, leveraging cross-modality transfer learning from an existing CT segmentation model to segment 40 anatomical structures. The annotation process began with a model based on transfer learning between CT and MR, which was iteratively refined based on manual corrections to predicted segmentations. The model's performance was evaluated on MRI examinations obtained from the German National Cohort (NAKO) study (n=900) from the AMOS22 dataset (n=60) and from the TotalSegmentator-MRI test data (n=29). The Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD) were used to assess segmentation quality, stratified by organ and scan type. The model and its weights will be open-sourced.
Results: MRSegmentator demonstrated high accuracy for well-defined organs (lungs: DSC 0.96, heart: DSC 0.94) and organs with anatomic variability (liver: DSC 0.96, kidneys: DSC 0.95). Smaller structures showed lower accuracy (portal/splenic veins: DSC 0.64, adrenal glands: DSC 0.69). On external validation using NAKO data, mean DSC ranged from 0.85 $\pm$ 0.08 for T2-HASTE to 0.91 $\pm$ 0.05 for in-phase sequences. The model generalized well to CT, achieving mean DSC of 0.84 $\pm$ 0.11 on AMOS CT data.
Conclusion: MRSegmentator accurately segments 40 anatomical structures in MRI across diverse datasets and imaging protocols, with additional generalizability to CT images. This open-source model will provide a valuable tool for automated multi-organ segmentation in medical imaging research. It can be downloaded from https://github.com/hhaentze/MRSegmentator.
MRAnnotator: multi-Anatomy and many-Sequence MRI segmentation of 44 structures
BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院生物医学工程与影像研究所）；Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院诊断、分子与介入放射学系）；Rotterdam School of Management, Erasmus University Rotterdam（伊拉斯姆斯大学鹿特丹管理学院）；East River Medical Imaging（东河医学影像中心）；Department of Radiology, Cornell Medicine（康奈尔医学院放射学系）；Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院温赖克人工智能与人类健康系）
Improve Cross-Modality Segmentation by Treating T1-Weighted MRI Images as Inverted CT Scans
Charité - Universitätsmedizin Berlin（柏林查理医院）；Freie Universität Berlin（柏林自由大学）；Humboldt Universität zu Berlin（柏林洪堡大学）；Radboud University Medical Center（拉德伯德大学医学中心）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）；Technical University of Munich（慕尼黑技术大学）
Cross-modality image estimation involves the generation of images of one medical imaging modality from that of another modality. Convolutional neural networks (CNNs) have been shown to be useful in identifying, characterising and extracting image patterns. Generative adversarial networks (GANs) use CNNs as generators and estimated images are discriminated as true or false based on an additional network. CNNs and GANs within the image estimation framework may be considered more generally as deep learning approaches, since imaging data tends to be large, leading to a larger number of network wei…
VRVis Zentrum für Virtual Reality und Visualisierung Forschungs-GmbH（VRVis虚拟现实与可视化研究有限公司）；Center for Medical Physics and Biomedical Engineering, Medical University of Vienna（维也纳医科大学医学物理与生物医学工程中心）；Center for Nuclear Medicine, Clinical Center of Serbia（塞尔维亚临床中心核医学中心）
Tumor segmentation in multimodal medical images has seen a growing trend towards deep learning based methods. Typically, studies dealing with this topic fuse multimodal image data to improve the tumor segmentation contour for a single imaging modality. However, they do not take into account that tumor characteristics are emphasized differently by each modality, which affects the tumor delineation. Thus, the tumor segmentation is modality- and task-dependent. This is especially the case for soft tissue sarcomas, where, due to necrotic tumor tissue, the segmentation differs vastly. Closing this …
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial People’s Hospital（广东省人民医院）；Guangdong Academy of Medical Sciences（广东省医学科学院）
Brain tumor segmentation based on multi-modal magnetic resonance imaging (MRI) plays a pivotal role in assisting brain cancer diagnosis, treatment, and postoperative evaluations. Despite the achieved inspiring performance by existing automatic segmentation methods, multi-modal MRI data are still unavailable in real-world clinical applications due to quite a few uncontrollable factors (e.g. different imaging protocols, data corruption, and patient condition limitations), which lead to a large performance drop during practical applications. In this work, we propose a Deeply supervIsed knowledGE …
Shanghai Jiao Tong University（上海交通大学）；Shandong Cancer Hospital Affiliated to Shandong University（山东大学附属山东省肿瘤医院）；Deepwise AI Lab（深睿医疗人工智能实验室）
Segmentation of nasopharyngeal carcinoma (NPC) from Magnetic Resonance Images (MRI) is a crucial prerequisite for NPC radiotherapy. However, manually segmenting of NPC is time-consuming and labor-intensive. Additionally, single-modality MRI generally cannot provide enough information for its accurate delineation. Therefore, a multi-modality MRI fusion network (MMFNet) based on three modalities of MRI (T1, T2 and contrast-enhanced T1) is proposed to complete accurate segmentation of NPC. The backbone of MMFNet is designed as a multi-encoder-based network, consisting of several encoders to captu…
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）
In this paper, the problem of orientation correction in cardiac MRI images is investigated and a framework for orientation recognition via deep neural networks is proposed. For multi-modality MRI, we introduce a transfer learning strategy to transfer our proposed model from single modality to multi-modality. We embed the proposed network into the orientation correction command-line tool, which can implement orientation correction on 2D DICOM and 3D NIFTI images. Our source code, network models and tools are available at https://github.com/Jy-stdio/MSCMR_orient/
Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
Our contribution is a unified cross-modality feature disentagling approach for multi-domain image translation and multiple organ segmentation. Using CT as the labeled source domain, our approach learns to segment multi-modal (T1-weighted and T2-weighted) MRI having no labeled data. Our approach uses a variational auto-encoder (VAE) to disentangle the image content from style. The VAE constrains the style feature encoding to match a universal prior (Gaussian) that is assumed to span the styles of all the source and target modalities. The extracted image style is converted into a latent style sc…
Shandong Normal University（山东师范大学）；Western University（韦仕敦大学）
Simultaneous segmentation and detection of liver tumors (hemangioma and hepatocellular carcinoma (HCC)) by using multi-modality non-contrast magnetic resonance imaging (NCMRI) are crucial for the clinical diagnosis. However, it is still a challenging task due to: (1) the HCC information on NCMRI is invisible or insufficient makes extraction of liver tumors feature difficult; (2) diverse imaging characteristics in multi-modality NCMRI causes feature fusion and selection difficult; (3) no specific information between hemangioma and HCC on NCMRI cause liver tumors detection difficult. In this stu…
Cross-Modal Vertical Federated Learning for MRI Reconstruction
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent Jarvis Lab（腾讯Jarvis实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）
We developed a new joint probabilistic segmentation and image distribution matching generative adversarial network (PSIGAN) for unsupervised domain adaptation (UDA) and multi-organ segmentation from magnetic resonance (MRI) images. Our UDA approach models the co-dependency between images and their segmentation as a joint probability distribution using a new structure discriminator. The structure discriminator computes structure of interest focused adversarial loss by combining the generated pseudo MRI with probabilistic segmentations produced by a simultaneously trained segmentation sub-networ…
Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
Accurate and robust segmentation of lung cancers from CT, even those located close to mediastinum, is needed to more accurately plan and deliver radiotherapy and to measure treatment response. Therefore, we developed a new cross-modality educed distillation (CMEDL) approach, using unpaired CT and MRI scans, whereby an informative teacher MRI network guides a student CT network to extract features that signal the difference between foreground and background. Our contribution eliminates two requirements of distillation methods: (i) paired image sets by using an image to image (I2I) translation a…
Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；Guangxi Medical University（广西医科大学）；The Hong Kong University of Science and Technology（香港科技大学）
Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
Lung tumors, especially those located close to or surrounded by soft tissues like the mediastinum, are difficult to segment due to the low soft tissue contrast on computed tomography images. Magnetic resonance images contain superior soft-tissue contrast information that can be leveraged if both modalities were available for training. Therefore, we developed a cross-modality educed learning approach where MR information that is educed from CT is used to hallucinate MRI and improve CT segmentation. Our approach, called cross-modality educed deep learning segmentation (CMEDL) combines CT and pse…
Modality Contribution Score - A Per-Patient Framework for Quantifying the Relative Diagnostic Contribution of Structural MRI and Amyloid PET in Alzheimer's Disease
Taipei Medical University（台北医学大学）；University of Memphis（孟菲斯大学）；Chitkara University（奇特卡拉大学）；University College London（伦敦大学学院）；UPES（石油与能源研究大学）；University of Eastern Finland（东芬兰大学）；CNRS（法国国家科学研究中心）；University Bordeaux（波尔多大学）；Bordeaux INP（波尔多国立理工学院）
University of Florida（佛罗里达大学）；Georgia State University（佐治亚州立大学）
Generating multi-contrasts/modal MRI of the same anatomy enriches diagnostic information but is limited in practice due to excessive data acquisition time. In this paper, we propose a novel deep-learning model for joint reconstruction and synthesis of multi-modal MRI using incomplete k-space data of several source modalities as inputs. The output of our model includes reconstructed images of the source modalities and high-quality image synthesized in the target modality. Our proposed model is formulated as a variational problem that leverages several learnable modality-specific feature extract…
Mamba-Based Modality Disentanglement Network for Multi-Contrast MRI Reconstruction
School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）；School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）
Automated Landmark Detection for assessing hip conditions: A Cross-Modality Validation of MRI versus X-ray
MaLGa Center, DIBRIS University of Genoa（MaLGa中心，热那亚大学DIBRIS学院）；Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences University of Oxford（牛津大学风湿病学与肌肉骨骼科学系）；Department of Computer Science University of Oxford（牛津大学计算机科学系）
Memorial Sloan-Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
Lack of large expert annotated MR datasets makes training deep learning models difficult. Therefore, a cross-modality (MR-CT) deep learning segmentation approach that augments training data using pseudo MR images produced by transforming expert-segmented CT images was developed. Eighty-One T2-weighted MRI scans from 28 patients with non-small cell lung cancers were analyzed. Cross-modality prior encoding the transformation of CT to pseudo MR images resembling T2w MRI was learned as a generative adversarial deep learning model. This model augmented training data arising from 6 expert-segmented …
Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequences
Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine（首尔大学医学院癌症生物学跨学科项目）；Department of Radiology, Seoul National University Hospital（首尔大学医院放射科）；Department of Radiology, Seoul National University College of Medicine（首尔大学医学院放射科）；Healthcare AI Research Institute, Seoul National University Hospital（首尔大学医院医疗人工智能研究所）
Label-Efficient Cross-Modality Generalization for Liver Segmentation in Multi-Phase MRI
Ho Chi Minh University of Science（胡志明科技大学）；National Central University（国立中央大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Houston（休斯顿大学）；Northwestern University（西北大学）
Deep Unfolding Network with Spatial Alignment for multi-modal MRI reconstruction
Shanghai University（上海大学）；School of Science, Shanghai University（上海大学理学院）；School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）；Shanghai Institute of Applied Mathematics and Mechanics, Shanghai University（上海大学上海应用数学与力学研究所）；School of Mechanics and Engineering Science, Shanghai University（上海大学力学与工程科学学院）
Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training
Department of Engineering Science, University of Oxford, Oxford, UK（工程科学系，牛津大学，牛津，英国）；Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK（牛津大学临床神经科学系，牛津，英国）
Anatomy-Informed Deep Learning and Radiomics for Automated Neurofibroma Segmentation in Whole-Body MRI
Institute for Applied Medical Informatics（应用医学信息学研究所）；Institute of Computational Neuroscience（计算神经科学研究所）；Center for Biomedical Artificial Intelligence (bAIome)（生物医学人工智能中心）；University Medical Center Hamburg-Eppendorf（汉堡埃彭多夫大学医学中心）；Department of Diagnostic and Interventional Radiology and Nuclear Medicine（诊断与介入放射学及核医学部门）；Department of Neurology（神经内科部门）
MMR-Mamba: Multi-Modal MRI Reconstruction with Mamba and Spatial-Frequency Information Fusion
The Hong Kong Polytechnic University（香港理工大学）；University of Science and Technology of China（中国科学技术大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Stanford University（斯坦福大学）
Shanghai Jiao Tong University（上海交通大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能科技有限公司）；ShanghaiTech University（上海科技大学）
In clinical practice, multi-modal magnetic resonance imaging (MRI) with different contrasts is usually acquired in a single study to assess different properties of the same region of interest in the human body. The whole acquisition process can be accelerated by having one or more modalities under-sampled in the $k$-space. Recent research has shown that, considering the redundancy between different modalities, a target MRI modality under-sampled in the $k$-space can be more efficiently reconstructed with a fully-sampled reference MRI modality. However, we find that the performance of the afore…
Deep Learning is a promising approach to either automate or simplify several tasks in the healthcare domain. In this work, we introduce SegAN-CAT, an approach to brain tumor segmentation in Magnetic Resonance Images (MRI), based on Adversarial Networks. In particular, we extend SegAN, successfully applied to the same task in a previous work, in two respects: (i) we used a different model input and (ii) we employed a modified loss function to train the model. We tested our approach on two large datasets, made available by the Brain Tumor Image Segmentation Benchmark (BraTS). First, we trained a…
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
Due to the difficulties of obtaining multimodal paired images in clinical practice, recent studies propose to train brain tumor segmentation models with unpaired images and capture complementary information through modality translation. However, these models cannot fully exploit the complementary information from different modalities. In this work, we thus present a novel two-step (intra-modality and inter-modality) curriculum disentanglement learning framework to effectively utilize privileged semi-paired images, i.e. limited paired images that are only available in training, for brain tumor …
East China Normal University（华东师范大学）；Tongji University（同济大学）
The encoder-decoder networks are commonly used in medical image segmentation due to their remarkable performance in hierarchical feature fusion. However, the expanding path for feature decoding and spatial recovery does not consider the long-term dependency when fusing feature maps from different layers, and the universal encoder-decoder network does not make full use of the multi-modality information to improve the network robustness especially for segmenting medical MRI. In this paper, we propose a novel feature fusion unit called Recurrent Decoding Cell (RDC) which leverages convolutional R…
Accelerated Multi-Contrast MRI Reconstruction via Frequency and Spatial Mutual Learning
University of Science and Technology of China（中国科学技术大学）；Stanford University（斯坦福大学）；Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（香港智能科学研究所-人工智能与机器人中心（CAIR））
DuDoUniNeXt: Dual-domain unified hybrid model for single and multi-contrast undersampled MRI reconstruction
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（中国科学技术大学生命科学与医学部生物医学工程学院）；Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advance Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院医学影像、机器人、分析计算与学习中心）；School of Electrical Engineering and Computer Science, The University of Queensland（昆士兰大学电气工程与计算机科学学院）
In most medical image processing tasks, the orientation of an image would affect computing result. However, manually reorienting images wastes time and effort. In this paper, we study the problem of recognizing orientation in cardiac MRI and using deep neural network to solve this problem. For multiple sequences and modalities of MRI, we propose a transfer learning strategy, which adapts our proposed model from a single modality to multiple modalities. We also propose a prediction method that uses voting. The results shows that deep neural network is an effective way in recognition of cardiac …
Completed Feature Disentanglement Learning for Multimodal MRIs Analysis
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Medical School of Tianjin University（天津大学医学院）；Peng Cheng Lab（鹏城实验室）；The University of Hong Kong（香港大学）；Tianjin Huanhu Hospital（天津环湖医院）
COSMOS: Cross-Modality Unsupervised Domain Adaptation for 3D Medical Image Segmentation based on Target-aware Domain Translation and Iterative Self-Training
Backbone-Conditional Behavior of Modality Gating in Multi-Modal Prostate MRI Segmentation: A 5-Fold Cross-Validation and Gate Mechanism Analysis
The Second Xiangya Hospital of Central South University（中南大学湘雅医学院第二医院）；The Third Xiangya Hospital of Central South University（中南大学湘雅医学院第三医院）；School of Life Sciences, Central South University（中南大学生命科学学院）；Hunan Provincial Key Laboratory of Medical Information Research (Central South University)（湖南省医学信息研究重点实验室（中南大学））；Hunan Provincial Clinical Medical Research Center for Cardiovascular Intelligent Medicine（湖南省心血管智能医学临床医学研究中心）
Towards Personalized Multi-Modal MRI Synthesis across Heterogeneous Datasets
University of Science and Technology of China（中国科学技术大学）；University of Electronic Science and Technology of China（电子科技大学）；Capital Medical University（首都医科大学）；Army Medical University（中国人民解放军陆军军医大学）；China National Clinical Research Center for Neurological Diseases（中国国家神经疾病临床研究中心）
Collaborative Attention and Consistent-Guided Fusion of MRI and PET for Alzheimer's Disease Diagnosis
School of Software Yunnan University（软件学院 云南大学）；Department of Computer Science The University of Sheffield（计算机科学系 剑桥大学）；Department of Computing Xian JiaoTong-Liverpool University（计算系 西交利物浦大学）
University College London（伦敦大学学院）；Biomediq A/S（Biomediq公司）；University of Copenhagen（哥本哈根大学）
Segmenting vascular pathologies such as white matter lesions in Brain magnetic resonance images (MRIs) require acquisition of multiple sequences such as T1-weighted (T1-w) --on which lesions appear hypointense-- and fluid attenuated inversion recovery (FLAIR) sequence --where lesions appear hyperintense--. However, most of the existing retrospective datasets do not consist of FLAIR sequences. Existing missing modality imputation methods separate the process of imputation, and the process of segmentation. In this paper, we propose a method to link both modality imputation and segmentation using…
