一种结合Mamba-Transformer网络的多模态特征蒸馏用于缺失模态的脑肿瘤分割
A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities
AI总结：
本文提出一种结合Mamba-Transformer网络的多模态特征蒸馏方法，用于在缺失模态情况下实现更准确的脑肿瘤分割。
摘要：
现有的脑肿瘤分割方法通常利用多种磁共振成像（MRI）模态在脑肿瘤图像中进行分割，可以实现更好的分割性能。然而，在临床应用中，由于资源限制，某些模态常常缺失，导致依赖完整模态分割的方法性能显著下降。本文提出了一种多模态特征蒸馏与Mamba-Transformer混合网络（MMTSeg）用于准确的脑肿瘤分割，即使模态缺失。我们首先采用多模态特征蒸馏（MFD）模块将特征级多模态知识蒸馏到不同的单模态中，以提取完整的模态信息。我们进一步开发了单模态特征增强（UFE）模块，以建模全局和局部信息之间的语义关系。最后，我们构建了一个跨模态融合（CMF）模块，以显式对齐不同模态之间的全局相关性，即使某些模态缺失。互补的特征在模态内和模态间通过UFE和CMF模块中的Mamba-Transformer混合架构进行精炼，动态捕捉长距离依赖和全局语义信息，以处理复杂的空间上下文。采用边界损失函数作为所提MMTSeg的分割损失，以最小化边界差异，以距离度量为基础。我们的消融研究证明了所提特征增强和融合模块在所提网络和Transformer与Mamba块中的重要性，对于改进缺失模态下的脑肿瘤分割性能具有重要作用。在BraTS 2018和BraTS 2020数据集上的广泛实验表明，所提MMTSeg框架在模态缺失时优于现有最先进方法。
英文摘要：
Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentation, which can achieve better segmentation performance. However, in clinical applications, some modalities are often missing due to resource constraints, resulting in significant performance degradation for methods that rely on complete modality segmentation. In this paper, we propose a Multimodal feature distillation with Mamba-Transformer hybrid network (MMTSeg) for accurate brain tumor segmentation with missing modalities. We first employ a Multimodal Feature Distillation (MFD) module to distill feature-level multimodal knowledge into different unimodalities to extract complete modality information. We further develop an Unimodal Feature Enhancement (UFE) module to model the semantic relationship between global and local information. Finally, we built a Cross-Modal Fusion (CMF) module to explicitly align the global correlations across modalities, even when some modalities are missing. Complementary features within and across modalities are refined by the Mamba-Transformer hybrid architectures in both the UFE and CMF modules, dynamically capturing long-range dependencies and global semantic information for complex spatial contexts. A boundary-wise loss function is employed as the segmentation loss of the proposed MMTSeg to minimize boundary discrepancies for a distance-based metric. Our ablation study demonstrates the importance of the proposed feature enhancement and fusion modules in the proposed network and the Transformer with Mamba block for improving the performance of brain tumor segmentation with missing modalities. Extensive experiments on the BraTS 2018 and BraTS 2020 datasets demonstrate that the proposed MMTSeg framework outperforms state-of-the-art methods when modalities are missing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
