arXiv 2501.157982025-01-27cs.CV

MM-Retinal V2：将精英知识火种迁移至眼底视觉-语言预训练

MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining

School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院眼科部）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research（科技研究局高性能计算研究所）

Ruiqi Wu, Na Su, Chenran Zhang, Tengfei Ma, Tao Zhou, Zhiting Cui, Nianfeng Tang, Tianyu Mao, Yi Zhou, Wen Fan, Tianxing Wu, Shenqi Jing, Huazhu Fu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对眼底VLP依赖大规模私有数据且预训练方式受限的问题，提出含CFP等模态的MM-Retinal V2数据集与KeepFIT V2模型，通过整合精英知识、结合对比与生成学习，在多设置下性能媲美顶尖私有数据模型且公开可用。

摘要：

视觉-语言预训练（VLP）已被用于在眼底图像分析的多种下游任务中实现泛化。尽管近期方法取得了有前景的成果，但它们严重依赖大规模私有图文数据，却较少关注预训练方式，这限制了其进一步发展。本研究中，我们引入MM-Retinal V2，一个包含CFP、FFA和OCT图像模态的高质量图文配对数据集。然后，我们提出一种新型眼底视觉-语言预训练模型KeepFIT V2，该模型通过将精英数据火种的知识整合到分类公开数据集中进行预训练。具体而言，采用初步文本预训练为文本编码器配备基础眼科文本知识；此外，设计混合图文知识注入模块用于知识迁移，该模块本质上结合了对比学习的全局语义概念与生成学习的局部外观细节。在零样本、少样本和线性探测设置下的大量实验表明，KeepFIT V2具有良好的泛化性和可迁移性，其性能可与基于大规模私有图文数据集训练的最先进眼底VLP模型相媲美。我们的数据集和模型可通过https://github.com/lxirich/MM-Retinal公开获取。

英文摘要：

Vision-language pretraining (VLP) has been investigated to generalize across diverse downstream tasks for fundus image analysis. Although recent methods showcase promising achievements, they significantly rely on large-scale private image-text data but pay less attention to the pretraining manner, which limits their further advancements. In this work, we introduce MM-Retinal V2, a high-quality image-text paired dataset comprising CFP, FFA, and OCT image modalities. Then, we propose a novel fundus vision-language pretraining model, namely KeepFIT V2, which is pretrained by integrating knowledge from the elite data spark into categorical public datasets. Specifically, a preliminary textual pretraining is adopted to equip the text encoder with primarily ophthalmic textual knowledge. Moreover, a hybrid image-text knowledge injection module is designed for knowledge transfer, which is essentially based on a combination of global semantic concepts from contrastive learning and local appearance details from generative learning. Extensive experiments across zero-shot, few-shot, and linear probing settings highlight the generalization and transferability of KeepFIT V2, delivering performance competitive to state-of-the-art fundus VLP models trained on large-scale private image-text datasets. Our dataset and model are publicly available via https://github.com/lxirich/MM-Retinal.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。