arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.11231cs.CV
收藏

KPL:视觉语言模型的无训练医学知识挖掘

KPL: Training-Free Medical Knowledge Mining of Vision-Language Models

Zhejiang University(浙江大学);ZJU-UIUC Institute(浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院);ZJU-Angelalign R&D Center for Intelligence Healthcare(浙江大学-时代天使智能医疗研发中心);Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence(浙江省医学影像人工智能重点实验室);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局前沿人工智能研究中心);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所)

Jiaxiang Liu, Tianxiang Hu, Jiawei Du, Ruiyuan Zhang, Joey Tianyi Zhou, Zuozhu Liu

AI总结:

针对CLIP在医学图像零样本分类中因类别表示不足和模态差距导致的性能问题,提出知识代理学习(KPL),通过文本代理优化和多模态代理学习挖掘CLIP知识,在医学和自然数据集上超越所有基线。

摘要:

视觉语言模型(如CLIP)凭借大规模图像-文本预训练在图像识别中表现出色。然而,将CLIP推理应用于零样本分类,尤其是医学图像诊断,面临挑战,原因在于:1)仅用单一类别名称表示图像类别存在不足;2)CLIP编码器生成的视觉与文本空间之间存在模态差距。尽管尝试利用大型语言模型丰富疾病描述,但缺乏类别特定知识往往导致性能不佳。此外,经验证据表明,现有的针对自然图像数据集的零样本图像分类代理学习方法在应用于医学数据集时表现出不稳定性。为应对这些挑战,我们引入了知识代理学习(KPL)来从CLIP中挖掘知识。KPL旨在利用CLIP的多模态理解能力,通过文本代理优化和多模态代理学习进行医学图像分类。具体而言,KPL从构建的知识增强库中检索与图像相关的知识描述,以丰富语义文本代理。随后,它利用输入图像和这些描述(经CLIP编码),稳定地生成多模态代理,从而提升零样本分类性能。在医学和自然图像数据集上进行的大量实验表明,KPL能够实现有效的零样本图像分类,优于所有基线方法。这些发现凸显了从CLIP中挖掘知识用于医学图像分类及更广泛领域的巨大潜力。

英文摘要:

Visual Language Models such as CLIP excel in image recognition due to extensive image-text pre-training. However, applying the CLIP inference in zero-shot classification, particularly for medical image diagnosis, faces challenges due to: 1) the inadequacy of representing image classes solely with single category names; 2) the modal gap between the visual and text spaces generated by CLIP encoders. Despite attempts to enrich disease descriptions with large language models, the lack of class-specific knowledge often leads to poor performance. In addition, empirical evidence suggests that existing proxy learning methods for zero-shot image classification on natural image datasets exhibit instability when applied to medical datasets. To tackle these challenges, we introduce the Knowledge Proxy Learning (KPL) to mine knowledge from CLIP. KPL is designed to leverage CLIP's multimodal understandings for medical image classification through Text Proxy Optimization and Multimodal Proxy Learning. Specifically, KPL retrieves image-relevant knowledge descriptions from the constructed knowledge-enhanced base to enrich semantic text proxies. It then harnesses input images and these descriptions, encoded via CLIP, to stably generate multimodal proxies that boost the zero-shot classification performance. Extensive experiments conducted on both medical and natural image datasets demonstrate that KPL enables effective zero-shot image classification, outperforming all baselines. These findings highlight the great potential in this paradigm of mining knowledge from CLIP for medical image classification and broader areas.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑