利用大语言模型生成的视觉概念增强疾病持续学习
Augmenting Continual Learning of Diseases with LLM-Generated Visual Concepts
AI总结：
提出利用大语言模型生成视觉概念作为语义指导，结合跨模态注意力模块，增强医学图像分类的持续学习性能，在多个数据集上达到最先进水平。
摘要：
持续学习对于医学图像分类系统适应动态演变的临床环境至关重要。多模态信息的整合可以显著增强图像类别的持续学习。然而，尽管现有方法确实利用了文本模态信息，但它们仅依赖于包含类别名称的简单模板，从而忽略了更丰富的语义信息。为了解决这些局限性，我们提出了一种新颖的框架，利用大语言模型（LLMs）生成的视觉概念作为判别性语义指导。我们的方法通过基于相似性的过滤机制动态构建视觉概念池，以防止冗余。然后，为了将这些概念整合到持续学习过程中，我们采用了一个跨模态图像-概念注意力模块，并配合注意力损失。通过注意力机制，该模块能够利用相关视觉概念的语义知识，生成具有类别代表性的融合特征用于分类。在医学和自然图像数据集上的实验表明，我们的方法达到了最先进的性能，证明了我们方法的有效性和优越性。我们将公开发布代码。
英文摘要：
Continual learning is essential for medical image classification systems to adapt to dynamically evolving clinical environments. The integration of multimodal information can significantly enhance continual learning of image classes. However, while existing approaches do utilize textual modality information, they solely rely on simplistic templates with a class name, thereby neglecting richer semantic information. To address these limitations, we propose a novel framework that harnesses visual concepts generated by large language models (LLMs) as discriminative semantic guidance. Our method dynamically constructs a visual concept pool with a similarity-based filtering mechanism to prevent redundancy. Then, to integrate the concepts into the continual learning process, we employ a cross-modal image-concept attention module, coupled with an attention loss. Through attention, the module can leverage the semantic knowledge from relevant visual concepts and produce class-representative fused features for classification. Experiments on medical and natural image datasets show our method achieves state-of-the-art performance, demonstrating the effectiveness and superiority of our method. We will release the code publicly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
