具有遗忘抵抗性和病变意识的源无关域自适应视网膜图像分析与视觉-语言模型
Forgetting-Resistant and Lesion-Aware Source-Free Domain Adaptive Fundus Image Analysis with Vision-Language Model
AI总结：
本文提出FRLA方法，通过保留目标模型的自信预测和利用ViL模型的细粒度知识，提升视网膜图像诊断的源无关域自适应性能。
摘要：
源无关域自适应（SFDA）旨在将在源域训练的模型适应到目标域，仅使用未标记的目标域数据和源模型。考虑到传统SFDA方法在域偏移下不可避免地容易出错，最近更多的关注被投向了借助现成基础模型的SFDA，例如视觉-语言（ViL）模型。然而，现有利用ViL模型进行SFDA的研究面临两个问题：（i）虽然利用互信息来考虑ViL模型预测和目标模型之间的联合分布，但我们认为目标模型中一些优秀的预测仍然会出现遗忘，这表现在适应过程中某些类别的准确率下降；（ii）先前研究忽略了ViL模型中丰富的、细粒度的知识，这为视网膜图像诊断提供了详细的支撑。在本文中，我们介绍了一种新的具有遗忘抵抗性和病变意识（FRLA）的方法，用于利用ViL模型进行视网膜图像诊断的SFDA。具体而言，一个遗忘抵抗适应模块明确保留目标模型的自信预测，一个病变意识适应模块从ViL模型中生成块级预测，并利用它们帮助目标模型意识到病变区域并利用ViL模型的细粒度知识。大量实验表明，我们的方法不仅显著优于视觉-语言模型，而且在最先进方法上也实现了持续的改进。我们的代码将被发布。
英文摘要：
Source-free domain adaptation (SFDA) aims to adapt a model trained in the source domain to perform well in the target domain, with only unlabeled target domain data and the source model. Taking into account that conventional SFDA methods are inevitably error-prone under domain shift, recently greater attention has been directed to SFDA assisted with off-the-shelf foundation models, e.g., vision-language (ViL) models. However, existing works of leveraging ViL models for SFDA confront two issues: (i) Although mutual information is exploited to consider the joint distribution between the predictions of ViL model and the target model, we argue that the forgetting of some superior predictions of the target model still occurs, as indicated by the decline of the accuracies of certain classes during adaptation; (ii) Prior research disregards the rich, fine-grained knowledge embedded in the ViL model, which offers detailed grounding for fundus image diagnosis. In this paper, we introduce a novel forgetting-resistant and lesion-aware (FRLA) method for SFDA of fundus image diagnosis with ViL model. Specifically, a forgetting-resistant adaptation module explicitly preserves the confident predictions of the target model, and a lesion-aware adaptation module yields patch-wise predictions from ViL model and employs them to help the target model be aware of the lesion areas and leverage the ViL model's fine-grained knowledge. Extensive experiments show that our method not only significantly outperforms the vision-language model, but also achieves consistent improvements over the state-of-the-art methods. Our code will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
