放射学中视觉语言模型的视觉提示工程
Visual Prompt Engineering for Vision Language Models in Radiology
AI总结：
针对放射学零样本分类中CLIP模型缺乏局部病理关注的问题，提出在图像中嵌入箭头等视觉标记引导注意力，在四个胸部X射线数据集上将AUROC提升最多0.185，并增强可解释性。
摘要：
医学图像分类在临床决策中起着至关重要的作用，然而大多数模型局限于一组固定的预定义类别，限制了它们对新情况的适应性。对比语言-图像预训练（CLIP）通过多模态大规模预训练实现零样本分类，提供了一种有前景的解决方案。然而，尽管CLIP能有效捕捉全局图像内容，放射学需要更局部地关注特定病理区域，以增强可解释性和诊断准确性。为解决这一问题，我们探索了将视觉线索纳入零样本分类的潜力，将视觉标记（如箭头、边界框和圆圈）直接嵌入放射图像中以引导模型注意力。在四个公开的胸部X射线数据集上的评估表明，视觉标记将AUROC提高了最多0.185，凸显了它们在提升分类性能方面的有效性。此外，注意力图分析证实，视觉线索帮助模型聚焦于临床相关区域，从而产生更具可解释性的预测。为支持进一步研究，我们使用公开数据集，并在https://github.com/MIC-DKFZ/VPE-in-Radiology提供我们的代码库和预处理流程，作为医学影像中局部分类未来工作的参考点。
英文摘要：
Medical image classification plays a crucial role in clinical decision-making, yet most models are constrained to a fixed set of predefined classes, limiting their adaptability to new conditions. Contrastive Language-Image Pretraining (CLIP) offers a promising solution by enabling zero-shot classification through multimodal large-scale pretraining. However, while CLIP effectively captures global image content, radiology requires a more localized focus on specific pathology regions to enhance both interpretability and diagnostic accuracy. To address this, we explore the potential of incorporating visual cues into zero-shot classification, embedding visual markers, such as arrows, bounding boxes, and circles, directly into radiological images to guide model attention. Evaluating across four public chest X-ray datasets, we demonstrate that visual markers improve AUROC by up to 0.185, highlighting their effectiveness in enhancing classification performance. Furthermore, attention map analysis confirms that visual cues help models focus on clinically relevant areas, leading to more interpretable predictions.To support further research, we use public datasets and provide our codebase and preprocessing pipeline under https://github.com/MIC-DKFZ/VPE-in-Radiology, serving as a reference point for future work on localized classification in medical imaging.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
