MaskedCLIP：为半监督医学视觉-语言预训练桥接掩码空间与 CLIP 空间
MaskedCLIP: Bridging the Masked and CLIP Space for Semi-Supervised Medical Vision-Language Pre-training
AI总结：
本文提出半监督医学视觉-语言预训练框架 MaskedCLIP，通过桥接 Transformer 和掩码知识蒸馏联合利用配对与非配对图像数据，提升视网膜图像下游任务性能与数据效率。
摘要：
基础模型最近在医学图像分析领域获得了极大关注。最先进的方法要么通过视觉-语言预训练利用配对的图像-文本数据，要么通过自监督预训练利用非配对图像数据，以学习具备可泛化图像特征的基础模型，从而提升下游任务性能。然而，仅基于配对图像数据或非配对图像数据学习基础模型，会限制其学习更丰富、更全面图像特征的能力。本文研究了一个称为半监督视觉-语言预训练的新任务，旨在充分释放配对与非配对图像数据在基础模型学习中的潜力。为此，我们提出 MaskedCLIP，这是一种将掩码图像建模与对比语言-图像预训练协同结合的框架，用于半监督视觉-语言预训练。结合配对与非配对图像数据学习基础模型的关键挑战，在于由这两类数据得到的特征空间互不兼容。为解决该问题，我们提出使用一个 bridge transformer（桥接 Transformer）连接掩码特征空间与 CLIP 特征空间。通过这种方式，语义更具体的 CLIP 特征能够受益于更通用的掩码特征，以进行语义特征提取。我们进一步提出掩码知识蒸馏损失，将 CLIP 特征空间中原始图像特征的语义知识蒸馏回掩码特征空间中预测的掩码图像特征。借助这种相互交互的设计，我们的框架有效利用配对与非配对图像数据，为下游任务学习更具泛化性的图像特征。在视网膜图像分析上的大量实验证明了本方法的有效性和数据效率。
英文摘要：
Foundation models have recently gained tremendous popularity in medical image analysis. State-of-the-art methods leverage either paired image-text data via vision-language pre-training or unpaired image data via self-supervised pre-training to learn foundation models with generalizable image features to boost downstream task performance. However, learning foundation models exclusively on either paired or unpaired image data limits their ability to learn richer and more comprehensive image features. In this paper, we investigate a novel task termed semi-supervised vision-language pre-training, aiming to fully harness the potential of both paired and unpaired image data for foundation model learning. To this end, we propose MaskedCLIP, a synergistic masked image modeling and contrastive language-image pre-training framework for semi-supervised vision-language pre-training. The key challenge in combining paired and unpaired image data for learning a foundation model lies in the incompatible feature spaces derived from these two types of data. To address this issue, we propose to connect the masked feature space with the CLIP feature space with a bridge transformer. In this way, the more semantic specific CLIP features can benefit from the more general masked features for semantic feature extraction. We further propose a masked knowledge distillation loss to distill semantic knowledge of original image features in CLIP feature space back to the predicted masked image features in masked feature space. With this mutually interactive design, our framework effectively leverages both paired and unpaired image data to learn more generalizable image features for downstream tasks. Extensive experiments on retinal image analysis demonstrate the effectiveness and data efficiency of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
