arXiv 2602.236522026-03-03cs.CVcs.AI

面向MRI多器官异常检测的3D模态感知预训练

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection

School of Electronic, Electrical Engineering and Physics, Fujian University of Technology（福建工程学院电子电气工程学院）；School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院）；Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China（苏州中医医院影像科）

Haowen Zhu, Ning Yin, Xiaogen Zhou

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MedMAP框架，通过3D MRI的模态感知预训练提升多器官异常检测性能，实验表明其优于现有视觉-语言模型。

摘要：

视觉-语言模型（VLMs）在医学影像的复杂诊断任务中展现出强大潜力。然而，将其应用于多器官医学影像会引入两个主要挑战：（1）模态特定的视觉-语言对齐；（2）跨模态特征融合。在本工作中，我们提出MedMAP，一种医疗模态感知预训练框架，旨在增强3D MRI中的视觉-语言表示学习。MedMAP包括一个模态感知的视觉-语言对齐阶段和一个多器官异常检测的微调阶段。在预训练阶段，模态感知编码器隐式地捕捉联合模态分布，并改善视觉和文本表示之间的对齐。随后，我们对预训练的视觉编码器（同时冻结文本编码器）进行微调以适应下游任务。为此，我们整理了MedMoM-MRI3D数据集，包含7392个3D MRI体积-报告对，涵盖十二种MRI模态和九种异常，专门用于各种3D医学分析任务。在MedMoM-MRI3D上的广泛实验表明，MedMAP在基于3D MRI的多器官异常检测中显著优于现有VLMs。我们的代码可在https://github.com/RomantiDr/MedMAP上获得。

英文摘要：

Vision-language models (VLMs) show strong potential for complex diagnostic tasks in medical imaging. However, applying VLMs to multi-organ medical imaging introduces two principal challenges: (1) modality-specific vision-language alignment and (2) cross-modal feature fusion. In this work, we propose MedMAP, a Medical Modality-Aware Pretraining framework that enhances vision-language representation learning in 3D MRI. MedMAP comprises a modality-aware vision-language alignment stage and a fine-tuning stage for multi-organ abnormality detection. During the pre-training stage, the modality-aware encoders implicitly capture the joint modality distribution and improve alignment between visual and textual representations. We then fine-tune the pre-trained vision encoders (while keeping the text encoder frozen) for downstream tasks. To this end, we curated MedMoM-MRI3D, comprising 7,392 3D MRI volume-report pairs spanning twelve MRI modalities and nine abnormalities tailored for various 3D medical analysis tasks. Extensive experiments on MedMoM-MRI3D demonstrate that MedMAP significantly outperforms existing VLMs in 3D MRI-based multi-organ abnormality detection. Our code is available at https://github.com/RomantiDr/MedMAP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。