This study addresses the essential task of medical image segmentation, which involves the automatic identification and delineation of anatomical structures and pathological regions in medical images. Accurate segmentation is crucial in radiology, as it aids in the precise localization of abnormalities such as tumors, thereby enabling effective diagnosis, treatment planning, and monitoring of disease progression. Specifically, the size, shape, and location of tumors can significantly influence clinical decision-making and therapeutic strategies, making accurate segmentation a key component of radiological workflows. However, challenges posed by variations in MRI modalities, image artifacts, and the scarcity of labeled data complicate the segmentation task and impact the performance of traditional models. To overcome these limitations, we propose a novel self-supervised learning Multi-encoder nnU-Net architecture designed to process multiple MRI modalities independently through separate encoders. This approach allows the model to capture modality-specific features before fusing them for the final segmentation, thus improving accuracy. Our Multi-encoder nnU-Net demonstrates exceptional performance, achieving a Dice Similarity Coefficient (DSC) of 93.72%, which surpasses that of other models such as vanilla nnU-Net, SegResNet, and Swin UNETR. By leveraging the unique information provided by each modality, the model enhances segmentation tasks, particularly in scenarios with limited annotated data. Evaluations highlight the effectiveness of this architecture in improving tumor segmentation outcomes.
We apply an ensemble of modified TransBTS, nnU-Net, and a combination of both for the segmentation task of the BraTS 2021 challenge. In fact, we change the original architecture of the TransBTS model by adding Squeeze-and-Excitation blocks, an increasing number of CNN layers, replacing positional encoding in Transformer block with a learnable Multilayer Perceptron (MLP) embeddings, which makes Transformer adjustable to any input size during inference. With these modifications, we are able to largely improve TransBTS performance. Inspired by a nnU-Net framework we decided to combine it with our…
Southern Medical University（南方医科大学）；LaTIM（拉蒂姆实验室）；INSERM（法国国家健康与医学研究院）；Univ Brest（布雷斯特大学）；Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院）
Automatic segmentation of head and neck cancer (HNC) tumors and lymph nodes plays a crucial role in the optimization treatment strategy and prognosis analysis. This study aims to employ nnU-Net for automatic segmentation and radiomics for recurrence-free survival (RFS) prediction using pretreatment PET/CT images in multi-center HNC cohort. A multi-center HNC dataset with 883 patients (524 patients for training, 359 for testing) was provided in HECKTOR 2022. A bounding box of the extended oropharyngeal region was retrieved for each patient with fixed size of 224 x 224 x 224 $mm^{3}$. Then 3D nn…
Topology-Driven Fusion of nnU-Net and MedNeXt for Accurate Brain Tumor Segmentation on Sub-Saharan Africa Dataset
Institute of Engineering, Thapathali Campus, Nepal（尼泊尔工程学院，塔帕塔利校区）；Gandaki College of Engineering and Science, Pokhara University, Pokhara, Nepal（加德基工程与科学学院，博克拉大学，博克拉，尼泊尔）；Nepal Engineering College, Changunarayan-4, Bhaktapur, Nepal（尼泊尔工程学院，恰纳拉亚纳-4，巴赫塔普尔，尼泊尔）；Madan Bhandari University of Science and Technology, Chitlang, Nepal（马丹·班达里科学与技术大学，奇塔朗，尼泊尔）；Montreal Neurological Institute, McGill University, Montreal, QC, Canada（蒙特利尔神经研究所，麦吉尔大学，蒙特利尔，魁北克，加拿大）；Department of Biomedical Engineering, McGill University（麦吉尔大学生物医学工程系）；Medical Artificial Intelligence Laboratory (MAI Lab), Lagos, Nigeria（医学人工智能实验室（MAI实验室），拉各斯，尼日利亚）；Department of Physics, Federal University of Technology, Minna, Nigeria（物理系，联邦技术大学，米纳，尼日利亚）；Department of Computer Science, Johns Hopkins University（计算机科学系，约翰霍普金斯大学）；Department of Radiology and Radiological Science, Johns Hopkins School of Medicine（放射学与放射科学系，约翰霍普金斯医学院）；Nepal Applied Mathematics and Informatics Institute for Research (NAAMII), Nepal（尼泊尔应用数学与信息学研究所（NAAMII），尼泊尔）；Institute of Engineering, Purbanchal Campus, Nepal（尼泊尔工程学院，普尔班恰尔校区）
Frequency-Aware Ensemble Learning for BraTS 2025 Pediatric Brain Tumor Segmentation
Institute of Natural Sciences, School of Mathematical Sciences, MOE-LSC, Shanghai Jiao Tong University（自然科学院、数学科学学院、MOE-LSC、上海交通大学）；Department of Radiology, Huashan Hospital, Fudan University（放射科、华山医院、复旦大学）；Department of Neurosurgery, Huashan Hospital, Fudan University（神经外科、华山医院、复旦大学）
Motivated by the fact that forward and backward passes of a deep network naturally form symmetric mappings between input and output representations, we introduce a simple yet effective self-supervised vision model pretraining framework inspired by energy-based models (EBMs). In the proposed framework, we model energy estimation and data restoration as the forward and backward passes of a single network without any auxiliary components, e.g., an extra decoder. For the forward pass, we fit a network to an energy function that assigns low energy scores to samples that belong to an unlabeled datas…
German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））
The U-Net was presented in 2015. With its straight-forward and successful architecture it quickly evolved to a commonly used benchmark in medical image segmentation. The adaptation of the U-Net to novel problems, however, comprises several degrees of freedom regarding the exact architecture, preprocessing, training and inference. These choices are not independent of each other and substantially impact the overall performance. The present paper introduces the nnU-Net ('no-new-Net'), which refers to a robust and self-adapting framework on the basis of 2D and 3D vanilla U-Nets. We argue the stron…
Laboratory for Machine Perception and Learning（机器感知与学习实验室）；University of Central Florida（中佛罗里达大学）
In this paper, we review adversarial pretraining of self-supervised deep networks including both convolutional neural networks and vision transformers. Unlike the adversarial training with access to labeled examples, adversarial pretraining is complicated as it only has access to unlabeled examples. To incorporate adversaries into pretraining models on either input or feature level, we find that existing approaches are largely categorized into two groups: memory-free instance-wise attacks imposing worst-case perturbations on individual examples, and memory-based adversaries shared across examp…
KP Labs（KP实验室）；Institute of Theoretical and Applied Informatics, Polish Academy of Sciences（波兰科学院理论与应用信息学研究所）；Silesian University of Technology（西里西亚工业大学）
Cloud detection is a pivotal satellite image pre-processing step that can be performed both on the ground and on board a satellite to tag useful images. In the latter case, it can help to reduce the amount of data to downlink by pruning the cloudy areas, or to make a satellite more autonomous through data-driven acquisition re-scheduling of the cloudy areas. We approach this important task with nnU-Nets, a self-reconfigurable framework able to perform meta-learning of a segmentation network over various datasets. Our experiments, performed over Sentinel-2 and Landsat-8 multispectral images rev…
Helmholtz Imaging, German Cancer Research Center (DKFZ)（亥姆霍兹成像分部，德国癌症研究中心（DKFZ））；Division of Medical Image Computing, German Cancer Research Center (DKFZ)（医学图像计算分部，德国癌症研究中心（DKFZ））；Faculty of Mathematics and Computer Science, Heidelberg University（海德堡大学数学与计算机科学学院）；HIDSS4Health - Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院（HIDSS4Health））；Medical Faculty Heidelberg, Heidelberg University（海德堡大学海德堡医学院）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital（海德堡大学医院放射肿瘤科模式分析与学习组）
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）
We apply nnU-Net to the segmentation task of the BraTS 2020 challenge. The unmodified nnU-Net baseline configuration already achieves a respectable result. By incorporating BraTS-specific modifications regarding postprocessing, region-based training, a more aggressive data augmentation as well as several minor modifications to the nnUNet pipeline we are able to improve its segmentation performance substantially. We furthermore re-implement the BraTS ranking scheme to determine which of our nnU-Net variants best fits the requirements imposed by it. Our final ensemble took the first place in the…
OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks
ISIR, Sorbonne Université（索邦大学ISIR研究所）；Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague（布拉格捷克理工大学捷克信息学、机器人学与控制论研究所）；FEE CTU（捷克技术大学电气工程系）；CIIRC CTU（捷克技术大学捷克信息学、机器人学与控制论研究所）；Kyutai（Kyutai公司）；LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS（法国国家科学研究中心-古斯塔夫·埃菲尔大学巴黎路桥学院LIGM实验室）
IRSTEA（法国国家环境与农业科技研究院）；UMR TETIS（TETIS联合研究单位）；LIRMM（蒙彼利埃计算机科学与机器人实验室）；Univ. of Montpellier（蒙彼利埃大学）；CIRAD（法国国际农业发展研究中心）
Recurrent Neural Networks (RNNs) can be seriously impacted by the initial parameters assignment, which may result in poor generalization performances on new unseen data. With the objective to tackle this crucial issue, in the context of RNN based classification, we propose a new supervised layer-wise pretraining strategy to initialize network parameters. The proposed approach leverages a data-aware strategy that sets up a taxonomy of classification problems automatically derived by the model behavior. To the best of our knowledge, despite the great interest in RNN-based classification, this is…
nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Imaging（亥姆霍兹成像）；University of Heidelberg（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）
German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Helmholtz Imaging（亥姆霍兹成像中心）
Semantic segmentation is one of the most popular research areas in medical image computing. Perhaps surprisingly, despite its conceptualization dating back to 2018, nnU-Net continues to provide competitive out-of-the-box solutions for a broad variety of segmentation problems and is regularly used as a development framework for challenge-winning algorithms. Here we use nnU-Net to participate in the AMOS2022 challenge, which comes with a unique set of tasks: not only is the dataset one of the largest ever created and boasts 15 target structures, but the competition also requires submitted soluti…
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Washington（华盛顿大学）；City University of Hong Kong (Dongguan)（香港城市大学（东莞））；Medical Optical Technology R&D Center, Research Institute of Tsinghua（清华研究院医学光学技术研发中心）；Jinfeng Laboratory（金凤实验室）
Large curated datasets are necessary, but annotating medical images is a time-consuming, laborious, and expensive process. Therefore, recent supervised methods are focusing on utilizing a large amount of unlabeled data. However, to do so, is a challenging task. To address this problem, we propose a new 3D Cross-Pseudo Supervision (3D-CPS) method, a semi-supervised network architecture based on nnU-Net with the Cross-Pseudo Supervision method. We design a new nnU-Net based preprocessing. In addition, we set the semi-supervised loss weights to expand linearity with each epoch to prevent the mode…
Cerebral microbleeds are small hypointense lesions visible on magnetic resonance imaging (MRI) with gradient echo, T2*, or susceptibility weighted (SWI) imaging. Assessment of cerebral microbleeds is mostly performed by visual inspection. The past decade has seen the rise of semi-automatic tools to assist with rating and more recently fully automatic tools for microbleed detection. In this work, we explore the use of nnU-Net as a fully automated tool for microbleed segmentation. Data was provided by the ``Where is VALDO?'' challenge of MICCAI 2021. The final method consists of nnU-Net in the `…
We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-out patches in an input image, our method learns to select the correct patch, among other "distractor" patches sampled from the same image, to fill in the masked location. This classification objective sidesteps the need for predicting exact pixel values of the target patches. The…
++nnU-Net: Scaling nnU-Net with Prefix-Based Data Augmentation
Center Algoritmi / LASI, University of Minho, Braga, Portugal（阿尔戈里米中心/拉斯伊大学，明霍大学，布拉加，葡萄牙）；Institute for Artificial Intelligence in Medicine, University Medicine Essen, Essen, Germany（医学人工智能研究所，埃森医学院，埃森，德国）；Institute of Medical Informatics / Dept. of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Germany（医学信息学研究所/口腔和颅面外科部，亚琛大学医院，德国）；Faculty of Computer Science, University of Duisburg-Essen, Essen, Germany（计算机科学学院，杜伊斯堡-埃森大学，埃森，德国）
Phase Unwrapping of Color Doppler Echocardiography using Deep Learning
CREATIS（CREATIS实验室）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；INSA Lyon（里昂国立应用科学学院）；University of Lyon 1（里昂第一大学）；Institut Universitaire de France（法兰西大学研究院）
Auto-nnU-Net: Towards Automated Medical Image Segmentation
Institute of AI, Leibniz University Hannover（莱比锡大学人工智能研究所）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School（汉诺威医学院彼得·L·里赫特茨医学信息学研究所）；CAIMed: Lower Saxony Center for AI & Causal Methods in Medicine（下萨克森人工智能与因果方法医学中心）；L3S Research Center（L3S研究中心）
Federated nnU-Net for Privacy-Preserving Medical Image Segmentation
Artificial Intelligence in Medicine Laboratory (BCN-AIM)（人工智能医学实验室（BCN-AIM））；Departament de Matemàtiques i Informàtica（数学与计算机科学系）；Universitat de Barcelona（巴塞罗那大学）；Medical University of Gdańsk (GUMed)（格但斯克医学大学（GUMed））；Hospital Clínic de Barcelona (HCB)（巴塞罗那医院（HCB））；Lithuanian University of Health Sciences（立陶宛卫生科学大学）；Institució Catala（加泰罗尼亚机构）
Darmstadt University of Technology（达姆施塔特工业大学）；University Hospital Frankfurt（法兰克福大学医院）
Automatic segmentation of lung lesions in computer tomography has the potential to ease the burden of clinicians during the Covid-19 pandemic. Yet predictive deep learning models are not trusted in the clinical routine due to failing silently in out-of-distribution (OOD) data. We propose a lightweight OOD detection method that exploits the Mahalanobis distance in the feature space. The proposed approach can be seamlessly integrated into state-of-the-art segmentation pipelines without requiring changes in model architecture or training procedure, and can therefore be used to assess the suitabil…
German Cancer Research Center（德国癌症研究中心）；University of Heidelberg（海德堡大学）；DeepMind；Heidelberg University Hospital（海德堡大学医院）
Biomedical imaging is a driver of scientific discovery and core component of medical care, currently stimulated by the field of deep learning. While semantic segmentation algorithms enable 3D image analysis and quantification in many applications, the design of respective specialised solutions is non-trivial and highly dependent on dataset properties and hardware conditions. We propose nnU-Net, a deep learning framework that condenses the current domain knowledge and autonomously takes the key decisions required to transfer a basic architecture to different datasets and segmentation tasks. Wit…
Budget-Aware Uncertainty for Radiotherapy Segmentation QA Using nnU-Net
Department of Biomedical Sciences, Humanitas University（人类医学科学系，人类itas大学）；Radiotherapy and Radiosurgery Department, IRCCS Humanitas Research Hospital（放疗与放射外科部门，IRCCS人类itas研究医院）
Exploring Optical Flow Inclusion into nnU-Net Framework for Surgical Instrument Segmentation
Life and Health Sciences Research Institute (ICVS)（生命与健康科学研究所（ICVS））；School of Medicine, University of Minho（米尼奥大学医学院）；ICVS/3B’s - PT Government Associate Laboratory（ICVS/3B's - 葡萄牙政府附属实验室）；School of Technology, IPCA（IPCA 技术学院）；LASI – Associate Laboratory of Intelligent Systems（LASI——智能系统附属实验室）；Algoritmi Center, School of Engineering, University of Minho（米尼奥大学工程学院 Algoritmi 中心）；KARL STORZ SE & Co. KG（卡尔史托斯有限两合公司）
