具有领域不变多模态掩码重建的医学视觉-语言模型鲁棒预训练
Robust Pre-Training of Medical Vision-and-Language Models with Domain-Invariant Multi-Modal Masked Reconstruction
AI总结：
本文提出Robust-MMR框架，通过显式建模鲁棒性提升医学视觉-语言模型在跨领域和扰动下的表现，实现更可靠的医疗应用。
摘要：
医学视觉-语言模型在联合推理医学图像和临床文本方面显示出强大潜力，但其性能往往在因成像设备、获取协议和报告风格变化引起的领域偏移下下降。现有的多模态预训练方法大多忽视了鲁棒性，将其视为下游适应问题。在本工作中，我们提出了Robust Multi-Modal Masked Reconstruction（Robust-MMR），一种将鲁棒性目标显式纳入掩码视觉-语言学习的自监督预训练框架。Robust-MMR整合了不对称扰动感知掩码、领域一致性正则化和模态鲁棒性约束，以鼓励领域不变的表示。我们评估了Robust-MMR在多个医学视觉-语言基准上的性能，包括医学视觉问答（VQA-RAD、SLAKE、VQA-2019）、跨领域图像-文本分类（MELINDA）和鲁棒图像-描述检索（ROCO）。Robust-MMR在VQA-RAD上实现了78.9%的跨领域准确率，优于最强基线3.8个百分点，并在SLAKE和VQA-2019上分别达到74.6%和77.0%的准确率。在扰动评估下，Robust-MMR将VQA-RAD准确率从69.1%提升到75.6%。对于图像-文本分类，跨领域MELINDA准确率从70.3%提升到75.2%，而检索实验显示在扰动下平均排名退化从超过16降至4.1。定性结果进一步展示了疾病检测和结构异常评估的临床推理改进。这些发现表明，在预训练过程中显式建模鲁棒性可以产生更可靠且可转移的医学视觉-语言表示，适用于实际应用。
英文摘要：
Medical vision-language models show strong potential for joint reasoning over medical images and clinical text, but their performance often degrades under domain shift caused by variations in imaging devices, acquisition protocols, and reporting styles. Existing multi-modal pre-training methods largely overlook robustness, treating it as a downstream adaptation problem. In this work, we propose Robust Multi-Modal Masked Reconstruction (Robust-MMR), a self-supervised pre-training framework that explicitly incorporates robustness objectives into masked vision-language learning. Robust-MMR integrates asymmetric perturbation-aware masking, domain-consistency regularization, and modality-resilience constraints to encourage domain-invariant representations. We evaluate Robust-MMR on multiple medical vision-language benchmarks, including medical visual question answering (VQA-RAD, SLAKE, VQA-2019), cross-domain image-text classification (MELINDA), and robust image-caption retrieval (ROCO). Robust-MMR achieves 78.9% cross-domain accuracy on VQA-RAD, outperforming the strongest baseline by 3.8 percentage points, and reaches 74.6% and 77.0% accuracy on SLAKE and VQA-2019, respectively. Under perturbed evaluation, Robust-MMR improves VQA-RAD accuracy from 69.1% to 75.6%. For image-text classification, cross-domain MELINDA accuracy increases from 70.3% to 75.2%, while retrieval experiments show a reduction in mean rank degradation from over 16 to 4.1 under perturbation. Qualitative results further demonstrate improved clinical reasoning for disease detection and structural abnormality assessment. These findings show that explicitly modeling robustness during pre-training leads to more reliable and transferable medical vision-language representations for real-world deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
