视觉语言模型的知识基础适应策略：为住院医师培训构建独特的病例集用于筛查乳腺X光检查
Knowledge-grounded Adaptation Strategy for Vision-language Models: Building Unique Case-set for Screening Mammograms for Residents Training
AI总结：
本研究提出一种基于选择性采样和困难负样本挖掘的框架，将视觉语言模型适应到医学领域，在乳腺X光图像-文本检索任务中显著提升了Recall@K性能。
摘要：
在自然图像和文本对上预训练的视觉语言模型（VLM）由于领域偏移，在应用于医学场景时面临重大障碍。然而，将这些VLM适应或微调用于医学用途存在相当大的困难，包括领域不匹配、无法获取大规模数据集以及高度类别不平衡。因此，迫切需要有效的策略将这些VLM适应到医学领域，因为这种适应在医疗保健应用中将具有巨大价值。在本研究中，我们提出一个框架，旨在巧妙地将VLM定制到医学领域，采用选择性采样和困难负样本挖掘技术以提升检索任务的性能。我们通过在两个不同的VLM上实施该方法来验证其有效性：领域内VLM（MedCLIP）和领域外VLM（ALBEF）。我们使用两个包含乳腺X光图像及其对应报告的大规模数据集，评估这些模型在原始现成状态和经过我们提出的训练策略后的性能。我们的评估涵盖零样本、少样本和监督场景。通过我们的方法，我们观察到图像-文本检索任务的Recall@K性能有显著提升。
英文摘要：
A visual-language model (VLM) pre-trained on natural images and text pairs poses a significant barrier when applied to medical contexts due to domain shift. Yet, adapting or fine-tuning these VLMs for medical use presents considerable hurdles, including domain misalignment, limited access to extensive datasets, and high-class imbalances. Hence, there is a pressing need for strategies to effectively adapt these VLMs to the medical domain, as such adaptations would prove immensely valuable in healthcare applications. In this study, we propose a framework designed to adeptly tailor VLMs to the medical domain, employing selective sampling and hard-negative mining techniques for enhanced performance in retrieval tasks. We validate the efficacy of our proposed approach by implementing it across two distinct VLMs: the in-domain VLM (MedCLIP) and out-of-domain VLMs (ALBEF). We assess the performance of these models both in their original off-the-shelf state and after undergoing our proposed training strategies, using two extensive datasets containing mammograms and their corresponding reports. Our evaluation spans zero-shot, few-shot, and supervised scenarios. Through our approach, we observe a notable enhancement in Recall@K performance for the image-text retrieval task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
