基于增量成像模态的持续视网膜视觉-语言预训练
Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalities
AI总结：
提出首个眼底领域持续视觉-语言预训练框架RetCoP，通过排练策略和非对角信息蒸馏方法，将增量到达的多模态图像与文本特征整合入单一基础模型，有效缓解灾难性遗忘，实现最佳泛化与最低遗忘率。
摘要：
传统的眼底图像分析模型专注于单模态任务，忽略了眼底模态互补性，这限制了其通用性。最近，视网膜基础模型已经出现，但大多数仍然特定于模态。将多种眼底成像模态整合到单一基础模型中是有价值的。然而，在动态环境中，来自不同模态的数据通常增量到达，需要进行持续预训练。为了解决这个问题，我们提出了RetCoP，这是眼底领域的首个持续视觉-语言预训练框架，它将来自不同成像模态的图像和文本特征增量地整合到一个统一的基础模型中。为了缓解持续预训练中的灾难性遗忘，我们引入了一种利用代表性图像-文本对的排练策略和一种非对角信息蒸馏方法。前者允许模型回顾先前阶段的知识，而后者显式地保留了图像和文本表示之间的对齐。实验表明，RetCoP优于所有比较方法，实现了最佳的泛化和最低的遗忘率。代码可在 https://github.com/Yang-Yao/RetCoP 找到。
英文摘要：
Traditional fundus image analysis models focus on single-modal tasks, ignoring fundus modality complementarity, which limits their versatility. Recently, retinal foundation models have emerged, but most still remain modality-specific. Integrating multiple fundus imaging modalities into a single foundation model is valuable. However, in dynamic environments, data from different modalities often arrive incrementally, necessitating continual pre-training. To address this, we propose RetCoP, the first continual vision-language pre-training framework in the fundus domain, which incrementally integrates image and text features from different imaging modalities into a single unified foundation model. To mitigate catastrophic forgetting in continual pre-training, we introduce a rehearsal strategy utilizing representative image-text pairs and an off-diagonal information distillation approach. The former allows the model to revisit knowledge from previous stages, while the latter explicitly preserves the alignment between image and text representations. Experiments show that RetCoP outperforms all the compared methods, achieving the best generalization and lowest forgetting rate. The code can be found at https://github.com/Yuang-Yao/RetCoP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
