探索CLIP在糖尿病视网膜病变域泛化中的迁移学习能力
Exploring the Transfer Learning Capabilities of CLIP in Domain Generalization for Diabetic Retinopathy
AI总结：
针对糖尿病视网膜病变分类的域泛化难题，探究CLIP的迁移学习潜力，提出带可学习视觉令牌的上下文优化多模态微调策略，较基线F1分数提升1.8%，代码已开源。
摘要：
糖尿病视网膜病变（Diabetic Retinopathy，DR）是视力损伤的主要诱因，需要早期检测与治疗。开发鲁棒的DR分类AI模型具有巨大潜力，但核心挑战在于确保模型在数据分布各异的陌生域中具备泛化能力。针对这一问题，本文研究了DR分类场景下的跨域泛化，即域泛化（Domain Generalization，DG）。医学领域的DG问题本就颇具难度，而不同域（如患者人口统计学特征、疾病分期）的标注数据采集困难进一步加剧了这一挑战。近期部分研究已证实CLIP在处理自然图像DG问题上的有效性。本研究探究了CLIP的迁移学习能力，及其在DR分类跨域泛化中的潜力。我们开展了全面实验，评估CLIP解决DR分类DG问题的效果与潜力。此外，我们提出了一种名为带可学习视觉令牌的上下文优化（Context Optimization with Learnable Visual Tokens，CoOpLVT）的多模态微调策略，该策略通过以视觉特征为条件增强上下文优化效果。研究结果表明，所提方法较基线将F1分数提升了1.8%，凸显了其在DR分类有效DG方面的应用前景。我们的代码已公开于https://github.com/Sanoojan/CLIP-DRDG。
英文摘要：
Diabetic Retinopathy (DR), a leading cause of vision impairment, requires early detection and treatment. Developing robust AI models for DR classification holds substantial potential, but a key challenge is ensuring their generalization in unfamiliar domains with varying data distributions. To address this, our paper investigates cross-domain generalization, also known as domain generalization (DG), within the context of DR classification. DG, a challenging problem in the medical domain, is complicated by the difficulty of gathering labeled data across different domains, such as patient demographics and disease stages. Some recent studies have shown the effectiveness of using CLIP to handle the DG problem in natural images. In this study, we investigate CLIP's transfer learning capabilities and its potential for cross-domain generalization in diabetic retinopathy (DR) classification. We carry out comprehensive experiments to assess the efficacy and potential of CLIP in addressing DG for DR classification. Further, we introduce a multi-modal fine-tuning strategy named Context Optimization with Learnable Visual Tokens (CoOpLVT), which enhances context optimization by conditioning on visual features. Our findings demonstrate that the proposed method increases the F1-score by 1.8% over the baseline, thus underlining its promise for effective DG in DR classification. Our code is publicly available at https://github.com/Sanoojan/CLIP-DRDG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
