利用知识描述增强视觉语言模型的异常定位
Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions
AI总结：
本文提出一种利用分解医学知识增强视觉语言模型医学异常定位的方法，通过将医学概念拆解为基本属性与视觉模式，使0.23B Florence-2模型在仅用1.5%数据训练下即可媲美7B LLaVA模型性能。
摘要：
视觉语言模型（VLMs）在视觉定位任务中展现了令人印象深刻的能力。然而，它们在医学领域的有效性，特别是在医学图像内的异常检测和定位方面，仍未被充分探索。一个主要挑战是医学术语的复杂性和抽象性，这使得很难将病理异常术语与其对应的视觉特征直接关联起来。在这项工作中，我们引入了一种新方法，通过利用分解的医学知识来增强VLM在医学异常检测和定位中的性能。我们不是直接提示模型识别特定的异常，而是专注于将医学概念分解为基本属性和常见的视觉模式。这种策略促进了文本描述和视觉特征之间更强的对齐，改善了医学图像中异常的识别和定位。我们在0.23B Florence-2基础模型上评估了我们的方法，并证明它在异常定位方面取得了与显著更大的7B LLaVA医学VLMs相当的性能，尽管它仅在用于此类模型的1.5%的数据上进行训练。实验结果还证明了我们的方法在已知和以前未见的异常中的有效性，表明其具有强大的泛化能力。
英文摘要：
Visual Language Models (VLMs) have demonstrated impressive capabilities in visual grounding tasks. However, their effectiveness in the medical domain, particularly for abnormality detection and localization within medical images, remains underexplored. A major challenge is the complex and abstract nature of medical terminology, which makes it difficult to directly associate pathological anomaly terms with their corresponding visual features. In this work, we introduce a novel approach to enhance VLM performance in medical abnormality detection and localization by leveraging decomposed medical knowledge. Instead of directly prompting models to recognize specific abnormalities, we focus on breaking down medical concepts into fundamental attributes and common visual patterns. This strategy promotes a stronger alignment between textual descriptions and visual features, improving both the recognition and localization of abnormalities in medical images.We evaluate our method on the 0.23B Florence-2 base model and demonstrate that it achieves comparable performance in abnormality grounding to significantly larger 7B LLaVA-based medical VLMs, despite being trained on only 1.5% of the data used for such models. Experimental results also demonstrate the effectiveness of our approach in both known and previously unseen abnormalities, suggesting its strong generalization capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
