arXiv 2412.094022025-04-07cs.CV

MultiEYE：面向OCT增强的眼底图像视网膜疾病识别的数据集与基准

MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images

Hong Kong University of Science and Technology（香港科技大学）；Guangdong Weiren Meditech Co., Ltd.（广东伟仁医疗科技有限公司）；Yunnan United Vision Innovations Technology Co., Ltd.（云南联合视力创新科技有限公司）；Guangdong Hospital of Integrated Traditional Chinese and Western Medicine（广东省中西医结合医院）；Second People’s Hospital of Foshan（佛山市第二人民医院）

Lehan Wang, Chongchong Qi, Chubin Ou, Lin An, Mei Jin, Xiangbin Kong, Xiaomeng Li

备注：: Accepted at IEEE TMI 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出OCT增强的眼底图像疾病识别新设定，构建首个大规模多模态多类别眼病数据集MultiEYE，并设计OCT-CoDA概念蒸馏方法，通过可解释的跨模态知识迁移显著提升仅用眼底图像的诊断性能。

摘要：

现有的眼底图像与OCT图像多模态学习方法大多要求两种模态在训练和测试阶段均可用且严格配对，这在临床场景中实用性较低。为扩大临床应用范围，我们提出了一种新的设定——“OCT增强的眼底图像疾病识别”，该设定允许在训练阶段使用非配对的多模态数据，并在测试阶段仅依赖广泛可得的眼底照片。为对该设定进行基准评估，我们构建了首个面向眼病诊断的大规模多模态多类别数据集MultiEYE，并提出了一种OCT辅助的概念蒸馏方法（OCT-CoDA），该方法利用语义丰富的概念从OCT图像中提取疾病相关知识，并将其迁移到眼底模型中。具体而言，我们将图像-概念关系视为一种连接，用于将有用的知识从OCT教师模型蒸馏到眼底学生模型，从而显著提升基于眼底图像的诊断性能，并将跨模态知识迁移转化为可解释的过程。在多疾病分类任务上的大量实验表明，我们提出的OCT-CoDA展现出卓越的结果和可解释性，显示出巨大的临床应用潜力。我们的数据集和代码可在https://github.com/xmed-lab/MultiEYE获取。

英文摘要：

Existing multi-modal learning methods on fundus and OCT images mostly require both modalities to be available and strictly paired for training and testing, which appears less practical in clinical scenarios. To expand the scope of clinical applications, we formulate a novel setting, "OCT-enhanced disease recognition from fundus images", that allows for the use of unpaired multi-modal data during the training phase and relies on the widespread fundus photographs for testing. To benchmark this setting, we present the first large multi-modal multi-class dataset for eye disease diagnosis, MultiEYE, and propose an OCT-assisted Conceptual Distillation Approach (OCT-CoDA), which employs semantically rich concepts to extract disease-related knowledge from OCT images and leverage them into the fundus model. Specifically, we regard the image-concept relation as a link to distill useful knowledge from the OCT teacher model to the fundus student model, which considerably improves the diagnostic performance based on fundus images and formulates the cross-modal knowledge transfer into an explainable process. Through extensive experiments on the multi-disease classification task, our proposed OCT-CoDA demonstrates remarkable results and interpretability, showing great potential for clinical application. Our dataset and code are available at https://github.com/xmed-lab/MultiEYE.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。