面向跨模态医学影像-报告检索的掩码对比重建方法
Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrieval
AI总结：
针对跨模态医学影像-报告检索的模态异质性问题，提出以掩码数据为唯一输入的MCR预训练框架与MbA模态对齐策略，减少任务干扰、降低训练成本，在MIMIC-CXR数据集上取得最优检索性能。
摘要：
跨模态医学影像-报告检索任务在临床诊断及各类医学生成任务中发挥着重要作用，消除不同模态间的异质性以增强语义一致性是该任务的核心挑战。当前的视觉-语言预训练（Vision-Language Pretraining, VLP）模型将跨模态对比学习与掩码重建作为联合训练任务，可有效提升跨模态检索性能。这类框架通常采用双流输入，使用未掩码数据开展跨模态对比学习，掩码数据用于重建任务。但由于两个代理任务的输入存在显著差异，会引发任务竞争与信息干扰，模态内及跨模态特征的表示学习效果受到限制。本文提出一种高效的VLP框架，名为掩码对比与重建（Masked Contrastive and Reconstruction, MCR），该框架将掩码数据作为两个任务的唯一输入，既增强了任务间的关联，减少了二者的信息干扰与竞争，还能大幅降低所需的GPU显存与训练时间。此外，我们提出了一种名为聚合前映射（Mapping before Aggregation, MbA）的新型模态对齐策略。与现有方法不同，MbA在进行局部特征聚合前，先将不同模态映射到公共特征空间，从而减少细粒度语义信息的损失，助力提升模态对齐效果。我们在MIMIC-CXR数据集上开展了定性与定量实验，验证了所提方法的有效性，其在医学跨模态检索任务上取得了当前最优（state-of-the-art）性能。
英文摘要：
Cross-modal medical image-report retrieval task plays a significant role in clinical diagnosis and various medical generative tasks. Eliminating heterogeneity between different modalities to enhance semantic consistency is the key challenge of this task. The current Vision-Language Pretraining (VLP) models, with cross-modal contrastive learning and masked reconstruction as joint training tasks, can effectively enhance the performance of cross-modal retrieval. This framework typically employs dual-stream inputs, using unmasked data for cross-modal contrastive learning and masked data for reconstruction. However, due to task competition and information interference caused by significant differences between the inputs of the two proxy tasks, the effectiveness of representation learning for intra-modal and cross-modal features is limited. In this paper, we propose an efficient VLP framework named Masked Contrastive and Reconstruction (MCR), which takes masked data as the sole input for both tasks. This enhances task connections, reducing information interference and competition between them, while also substantially decreasing the required GPU memory and training time. Moreover, we introduce a new modality alignment strategy named Mapping before Aggregation (MbA). Unlike previous methods, MbA maps different modalities to a common feature space before conducting local feature aggregation, thereby reducing the loss of fine-grained semantic information necessary for improved modality alignment. Qualitative and quantitative experiments conducted on the MIMIC-CXR dataset validate the effectiveness of our approach, demonstrating state-of-the-art performance in medical cross-modal retrieval tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
