Diabetic retinopathy (DR) is a leading cause of preventable blindness worldwide, demanding accurate automated diagnostic systems. While general-domain vision-language models like Contrastive Language-Image Pre-Training (CLIP) perform well on natural image tasks, they struggle in medical domain applications, particularly in cross-modal retrieval for ophthalmological images. We propose a novel knowledge-enhanced joint embedding framework that integrates retinal fundus images, clinical text, and structured patient data through a multimodal transformer architecture to address the critical gap in medical image-text alignment. Our approach employs separate encoders for each modality: a Vision Transformer (ViT-B/16) for retinal images, Bio-ClinicalBERT for clinical narratives, and a multilayer perceptron for structured demographic and clinical features. These modalities are fused through a joint transformer with modality-specific embeddings, trained using multiple objectives including contrastive losses between modality pairs, reconstruction losses for images and text, and classification losses for DR severity grading according to ICDR and SDRG schemes. Experimental results on the Brazilian Multilabel Ophthalmological Dataset (BRSET) demonstrate significant improvements over baseline models. Our framework achieves near-perfect text-to-image retrieval performance with Recall@1 of 99.94% compared to fine-tuned CLIP's 1.29%, while maintaining state-of-the-art classification accuracy of 97.05% for SDRG and 97.97% for ICDR. Furthermore, zero-shot evaluation on the unseen DeepEyeNet dataset validates strong generalizability with 93.95% Recall@1 versus 0.22% for fine-tuned CLIP. These results demonstrate that our multimodal training approach effectively captures cross-modal relationships in the medical domain, establishing both superior retrieval capabilities and robust diagnostic performance.
Manipal Institute of Technology（马尼帕尔理工学院）；Manipal Academy of Higher Education（马尼帕尔高等教育学院）
Deep Ensemble Convolutional Neural Networks has become a methodology of choice for analyzing medical images with a diagnostic performance comparable to a physician, including the diagnosis of Diabetic Retinopathy. However, commonly used techniques are deterministic and are therefore unable to provide any estimate of predictive uncertainty. Quantifying model uncertainty is crucial for reducing the risk of misdiagnosis. A reliable architecture should be well-calibrated to avoid over-confident predictions. To address this, we propose a UATTA-ENS: Uncertainty-Aware Test-Time Augmented Ensemble Tec…
Montclair State University（蒙特克莱尔州立大学）；Austin Peay State University（奥斯汀佩伊州立大学）
Diabetic retinopathy (DR) is a retinal microvascular condition that emerges in diabetic patients. DR will continue to be a leading cause of blindness worldwide, with a predicted 191.0 million globally diagnosed patients in 2030. Microaneurysms, hemorrhages, exudates, and cotton wool spots are common signs of DR. However, they can be small and hard for human eyes to detect. Early detection of DR is crucial for effective clinical treatment. Existing methods to classify images require much time for feature extraction and selection, and are limited in their performance. Convolutional Neural Networ…
Manarat International University（玛纳拉特国际大学）；Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）
Diabetic Retinopathy (DR) is a constantly deteriorating disease, being one of the leading causes of vision impairment and blindness. Subtle distinction among different grades and existence of many significant small features make the task of recognition very challenging. In addition, the present approach of retinopathy detection is a very laborious and time-intensive task, which heavily relies on the skill of a physician. Automated detection of diabetic retinopathy is essential to tackle these problems. Early-stage detection of diabetic retinopathy is also very important for diagnosis, which ca…
From Retinal Pixels to Patients: Evolution of Deep Learning Research in Diabetic Retinopathy Screening
Fraunhofer IAIS - Department of Media Engineering, Germany（弗劳恩霍夫研究所媒体工程部门）；University of Bonn - Department of Computer Science, Germany（波恩大学计算机科学系）；University Hospital Bonn - Department of Ophthalmology, Germany（波恩大学医院眼科部门）；Lamarr Institute for Machine Learning and Artificial Intelligence, Germany（拉马尔人工智能与机器学习研究所）
Aalto University School of Science（阿尔托大学理学院）；Digifundus Ltd.（Digifundus有限公司）；Central Finland Central Hospital（中芬兰中心医院）
Diabetes is a globally prevalent disease that can cause visible microvascular complications such as diabetic retinopathy and macular edema in the human eye retina, the images of which are today used for manual disease screening. This labor-intensive task could greatly benefit from automatic detection using deep learning technique. Here we present a deep learning system that identifies referable diabetic retinopathy comparably or better than presented in the previous studies, although we use only a small fraction of images (<1/4) in training but are aided with higher image resolutions. We also …
University of Waterloo（滑铁卢大学）；University of Guelph（圭尔夫大学）；Canadian Institute of Advanced Research (CIFAR)（加拿大高等研究院）；Vector Institute（向量研究院）
Objective: Radiomics-driven Computer Aided Diagnosis (CAD) has shown considerable promise in recent years as a potential tool for improving clinical decision support in medical oncology, particularly those based around the concept of Discovery Radiomics, where radiomic sequencers are discovered through the analysis of medical imaging data. One of the main limitations with current CAD approaches is that it is very difficult to gain insight or rationale as to how decisions are made, thus limiting their utility to clinicians. Methods: In this study, we propose CLEAR-DR, a novel interpretable CAD …
COMSATS University Islamabad（伊斯兰堡 COMSATS 大学）；Deakin University（迪肯大学）；Victoria University（维多利亚大学）
Diabetic retinopathy refers to the pathology of the retina induced by diabetes and is one of the leading causes of preventable blindness in the world. Early detection of diabetic retinopathy is critical to avoid vision problem through continuous screening and treatment. In traditional clinical practice, the involved lesions are manually detected using photographs of the fundus. However, this task is cumbersome and time-consuming and requires intense effort due to the small size of lesion and low contrast of the images. Thus, computer-assisted diagnosis of diabetic retinopathy based on the dete…
Reliable microaneurysm detection in digital fundus images is still an open issue in medical image processing. We propose an ensemble-based framework to improve microaneurysm detection. Unlike the well-known approach of considering the output of multiple classifiers, we propose a combination of internal components of microaneurysm detectors, namely preprocessing methods and candidate extractors. We have evaluated our approach for microaneurysm detection in an online competition, where this algorithm is currently ranked as first and also on two other databases. Since microaneurysm detection is d…
Kasetsart University（泰国农业大学）；National Electronics and Computer Technology Centre（泰国国家电子与计算机技术中心）；Tokyo Institute of Technology（东京工业大学）
This research aims to develop an efficient system for screening of diabetic retinopathy. Diabetic retinopathy is the major cause of blindness. Severity of diabetic retinopathy is recognized by some features, such as blood vessel area, exudates, haemorrhages and microaneurysms. To grade the disease the screening system must efficiently detect these features. In this paper we are proposing a simple and fast method for detection of diabetic retinopathy. We do pre-processing of grey-scale image and find all labelled connected components (blobs) in an image regardless of whether it is haemorrhages,…
Although deep learning research and applications have grown rapidly over the past decade, it has shown limitation in healthcare applications and its reachability to people in remote areas. One of the challenges of incorporating deep learning in medical data classification or prediction is the shortage of annotated training data in the healthcare industry. Medical data sharing privacy issues and limited patient population size can be stated as some of the reasons for training data insufficiency in healthcare. Methods to exploit deep learning applications in healthcare have been proposed and imp…
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；University of California, Berkeley（加州大学伯克利分校）；Affiliated Eye Hospital of Nanchang University（南昌大学附属眼科医院）
Retinal vascular diseases affect the well-being of human body and sometimes provide vital signs of otherwise undetected bodily damage. Recently, deep learning techniques have been successfully applied for detection of diabetic retinopathy (DR). The main obstacle of applying deep learning techniques to detect most other retinal vascular diseases is the limited amount of data available. In this paper, we propose a transfer learning technique that aims to utilize the feature similarities for detecting retinal vascular diseases. We choose the well-studied DR detection as a source task and identify…
Optical Coherence Tomography Angiography-OCTA dataset for the study of Diabetic Retinopathy
Symbiosis Institute of Technology, Symbiosis International (Deemed University)（共生国际（ deemed 大学）共生技术学院）；University of Technology Sydney（悉尼科技大学）；Natasha Eye Care（娜塔莎眼科护理中心）
Sharda University（沙尔达大学）；Research and Technology Development Center（研究与技术开发中心）
Annotated training data insufficiency remains to be one of the challenges of applying deep learning in medical data classification problems. Transfer learning from an already trained deep convolutional network can be used to reduce the cost of training from scratch and to train with small training data for deep learning. This raises the question of whether we can use transfer learning to overcome the training data insufficiency problem in deep learning based medical data classifications. Deep convolutional networks have been achieving high performance results on the ImageNet Large Scale Visual…
University of Oxford（牛津大学）；Big Data Institute（大数据研究所）
Diabetic retinopathy (DR) is a complication of diabetes, and one of the major causes of vision impairment in the global population. As the early-stage manifestation of DR is usually very mild and hard to detect, an accurate diagnosis via eye-screening is clinically important to prevent vision loss at later stages. In this work, we propose an ensemble method to automatically grade DR using ultra-wide optical coherence tomography angiography (UW-OCTA) images available from Diabetic Retinopathy Analysis Challenge (DRAC) 2022. First, we adopt the state-of-the-art classification networks, i.e., Res…
UNIÃO DE ENSINO DO SUDOESTE DO PARANÁ – UNISEP（巴拉那州西南部教育联盟（UNISEP））；FACULDADE EDUCACIONAL DE FRANCISCO BELTRÃO – FEFB（弗朗西斯科贝尔特朗教育学院（FEFB））
Many people are affected by diabetes around the world. This disease may have type 1 and 2. Diabetes brings with it several complications including diabetic retinopathy, which is a disease that if not treated correctly can lead to irreversible damage in the patient's vision. The earlier it is detected, the better the chances that the patient will not lose vision. Methods of automating manual procedures are currently in evidence and the diagnostic process for retinopathy is manual with the physician analyzing the patient's retina on the monitor. The practice of image recognition can aid this det…
Shanghai University（上海大学）；Tongji University（同济大学）；Shanghai General Hospital（上海市第一人民医院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）
Diabetic retinopathy (DR) is one of the leading causes of blindness. However, no specific symptoms of early DR lead to a delayed diagnosis, which results in disease progression in patients. To determine the disease severity levels, ophthalmologists need to focus on the discriminative parts of the fundus images. In recent years, deep learning has achieved great success in medical image analysis. However, most works directly employ algorithms based on convolutional neural networks (CNNs), which ignore the fact that the difference among classes is subtle and gradual. Hence, we consider automatic …
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）
Convolutional neural networks (CNNs) are extensively beneficial for medical image processing. Medical images are plentiful, but there is a lack of annotated data. Transfer learning is used to solve the problem of lack of labeled data and grants CNNs better training capability. Transfer learning can be used in many different medical applications; however, the model under transfer should have the same size as the original network. Knowledge distillation is recently proposed to transfer the knowledge of a model to another one and can be useful to cover the shortcomings of transfer learning. But s…
Sun Yat-sen University（中山大学）；Southern University of Science and Technology（南方科技大学）
Manually annotating medical images is extremely expensive, especially for large-scale datasets. Self-supervised contrastive learning has been explored to learn feature representations from unlabeled images. However, unlike natural images, the application of contrastive learning to medical images is relatively limited. In this work, we propose a self-supervised framework, namely lesion-based contrastive learning for automated diabetic retinopathy (DR) grading. Instead of taking entire images as the input in the common contrastive learning scheme, lesion patches are employed to encourage the fea…
Cross Feature Fusion of Fundus Image and Generated Lesion Map for Referable Diabetic Retinopathy Classification
Sungkyunkwan University（成均馆大学）；College of Computing and Informatics, Sungkyunkwan University（成均馆大学计算与信息学院）；Dept. of Electrical and Computer Engineering, Sungkyunkwan University（成均馆大学电气与计算机工程系）
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Diabetic Retinopathy (DR) is a leading cause of vision loss globally. Yet despite its prevalence, the majority of affected people lack access to the specialized ophthalmologists and equipment required for assessing their condition. This can lead to delays in the start of treatment, thereby lowering their chances for a successful outcome. Machine learning systems that automatically detect the disease in eye fundus images have been proposed as a means of facilitating access to DR severity estimates for patients in remote regions or even for complementing the human expert's diagnosis. In this pap…
Replication studies are essential for validation of new methods, and are crucial to maintain the high standards of scientific publications, and to use the results in practice. We have attempted to replicate the main method in 'Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs' published in JAMA 2016; 316(22). We re-implemented the method since the source code is not available, and we used publicly available data sets. The original study used non-public fundus images from EyePACS and three hospitals in India for training.…
Artificial Intelligence Group, Baidu（百度人工智能组）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Science and Technology of China（中国科学技术大学）
Diabetic retinopathy (DR) grading from fundus images has attracted increasing interest in both academic and industrial communities. Most convolutional neural network (CNN) based algorithms treat DR grading as a classification task via image-level annotations. However, these algorithms have not fully explored the valuable information in the DR-related lesions. In this paper, we present a robust framework, which collaboratively utilizes patch-level and image-level annotations, for DR severity grading. By an end-to-end optimization, this framework can bi-directionally exchange the fine-grained le…
Robert Bosch GmbH（罗伯特·博世有限公司）；TU Dortmund University（多特蒙德工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Convolutional neural networks (CNNs) show impressive performance for image classification and detection, extending heavily to the medical image domain. Nevertheless, medical experts are sceptical in these predictions as the nonlinear multilayer structure resulting in a classification outcome is not directly graspable. Recently, approaches have been shown which help the user to understand the discriminative regions within an image which are decisive for the CNN to conclude to a certain class. Although these approaches could help to build trust in the CNNs predictions, they are only slightly sho…
Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology（香港科技大学）；Shanghai Jiao Tong University Affiliated Sixth People’s Hospital（上海交通大学附属第六人民医院）；VUNO Inc.（VUNO公司）；Johannes Kepler University Linz（林茨约翰·开普勒大学）；Fudan University（复旦大学）；Inserm（法国国家健康与医学研究院）；University of Western Brittany（西布列塔尼大学）
Commack High School（康马克高中）；Camp Illumina（Illumina 训练营）；Glenda Dawson High School（格伦达·道森高中）；Mira Loma High School（米拉洛马高中）；Ward Melville High School（沃德·梅尔维尔高中）
In this work, deep learning algorithms are used to classify fundus images in terms of diabetic retinopathy severity. Six different combinations of two model architectures, the Dense Convolutional Network-121 and the Residual Neural Network-50 and three image types, RGB, Green, and High Contrast, were tested to find the highest performing combination. We achieved an average validation loss of 0.17 and a max validation accuracy of 85 percent. By testing out multiple combinations, certain combinations of parameters performed better than others, though minimal variance was found overall. Green fil…
Cases of diabetes and related diabetic retinopathy (DR) have been increasing at an alarming rate in modern times. Early detection of DR is an important problem since it may cause permanent blindness in the late stages. In the last two decades, many different approaches have been applied in DR detection. Reviewing academic literature shows that deep neural networks (DNNs) have become the most preferred approach for DR detection. Among these DNN approaches, Convolutional Neural Network (CNN) models are the most used ones in the field of medical image classification. Designing a new CNN architect…
Universidad Nacional de Colombia（哥伦比亚国立大学）；Fundación Oftalmológica Nacional（国家眼科基金会）；Universidad del Rosario（罗萨里奥大学）
Diabetic retinopathy (DR) is the result of a complication of diabetes affecting the retina. It can cause blindness, if left undiagnosed and untreated. An ophthalmologist performs the diagnosis by screening each patient and analyzing the retinal lesions via ocular imaging. In practice, such analysis is time-consuming and cumbersome to perform. This paper presents a model for automatic DR classification on eye fundus images. The approach identifies the main ocular lesions related to DR and subsequently diagnoses the illness. The proposed method follows the same workflow as the clinicians, provid…
University of Bournemouth（伯恩茅斯大学）；École de Technolgie Superieure de Montréal（蒙特利尔高等技术学院）；Diagnos Inc.（Diagnos公司）；CRCHUM University of Montreal Hospital Centre（蒙特利尔大学医院研究中心）
Assessing the degree of disease severity in biomedical images is a task similar to standard classification but constrained by an underlying structure in the label space. Such a structure reflects the monotonic relationship between different disease grades. In this paper, we propose a straightforward approach to enforce this constraint for the task of predicting Diabetic Retinopathy (DR) severity from eye fundus images based on the well-known notion of Cost-Sensitive classification. We expand standard classification losses with an extra term that acts as a regularizer, imposing greater penaltie…
Heritage Institute of Technology（遗产理工学院）；Tata Consultancy Services（塔塔咨询服务公司）
Diabetic Retinopathy (DR) is a complication of long-standing, unchecked diabetes and one of the leading causes of blindness in the world. This paper focuses on improved and robust methods to extract some of the features of DR, viz. Blood Vessels and Exudates. Blood vessels are segmented using multiple morphological and thresholding operations. For the segmentation of exudates, k-means clustering and contour detection on the original images are used. Extensive noise reduction is performed to remove false positives from the vessel segmentation algorithm's results. The localization of Optic Disc …
Deep learning is quickly becoming the leading methodology for medical image analysis. Given a large medical archive, where each image is associated with a diagnosis, efficient pathology detectors or classifiers can be trained with virtually no expert knowledge about the target pathologies. However, deep learning algorithms, including the popular ConvNets, are black boxes: little is known about the local patterns analyzed by ConvNets to make a decision at the image level. A solution is proposed in this paper to create heatmaps showing which pixels in images play a role in the image-level predic…
Casey Eye Institute, Oregon Health & Science University（俄勒冈健康与科学大学凯西眼科研究所）；Department of Biomedical Engineering, Oregon Health & Science University（俄勒冈健康与科学大学生物医学工程系）
Objective: Optical coherence tomography (OCT) and its angiography (OCTA) have several advantages for the early detection and diagnosis of diabetic retinopathy (DR). However, automated, complete DR classification frameworks based on both OCT and OCTA data have not been proposed. In this study, a convolutional neural network (CNN) based method is proposed to fulfill a DR classification framework using en face OCT and OCTA. Methods: A densely and continuously connected neural network with adaptive rate dropout (DcardNet) is designed for the DR classification. In addition, adaptive label smoothing…
Detection and Classification of Diabetic Retinopathy using Deep Learning Algorithms for Segmentation to Facilitate Referral Recommendation for Test and Treatment Prediction
Southern University of Science and Technology（南方科技大学）；The University of British Columbia（不列颠哥伦比亚大学）；The University of Hong Kong（香港大学）；The University of Queensland（昆士兰大学）
Although deep learning based diabetic retinopathy (DR) classification methods typically benefit from well-designed architectures of convolutional neural networks, the training setting also has a non-negligible impact on the prediction performance. The training setting includes various interdependent components, such as objective function, data sampling strategy and data augmentation approach. To identify the key components in a standard deep learning framework (ResNet-50) for DR grading, we systematically analyze the impact of several major components. Extensive experiments are conducted on a …
Comprehensive Evaluation of OCT-based Automated Segmentation of Retinal Layer, Fluid and Hyper-Reflective Foci: Impact on Clinical Assessment of Diabetic Retinopathy Severity
School of Engineering Science, Simon Fraser University（西蒙弗雷泽大学工程科学学院）；School of Medicine, Wake Forest University（威克森林大学医学院）；Sankara Nethralaya（萨卡拉眼科医院）；Department of Ophthalmology and Visual Sciences, The University of British Columbia（不列颠哥伦比亚大学眼科与视觉科学系）；School of Biomedical Engineering, The University of British Columbia（不列颠哥伦比亚大学生物医学工程学院）；Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Department of Medical Physics and Biomedical Engineering, University College London（伦敦大学学院医学物理与生物医学工程系）；Department of Computer Science, Memorial University of Newfoundland（新斯科舍纪念大学计算机科学系）
PLADEMA Institute, UNICEN（国立中央布宜诺斯艾利斯省大学PLADEMA研究所）；Centro de Oftalmología Martínez（马丁内斯眼科中心）
Deep learning is currently the state-of-the-art for automated detection of referable diabetic retinopathy (DR) from color fundus photographs (CFP). While the general interest is put on improving results through methodological innovations, it is not clear how good these approaches perform compared to standard deep classification models trained with the appropriate settings. In this paper we propose to model a strong baseline for this task based on a simple and standard ResNet-18 architecture. To this end, we built on top of prior art by training the model with a standard preprocessing strategy …
Department of Computer Science & Engineering, Priyadarshini Institute of Engineering & Technology（普里达尔希尼工程技术学院计算机科学与工程系）
Diabetic Retinopathy is a critical health problem influences 100 million individuals worldwide, and these figures are expected to rise, particularly in Asia. Diabetic Retinopathy is a chronic eye disease which can lead to irreversible vision loss. Considering the visual complexity of retinal images, the early-stage diagnosis of Diabetic Retinopathy can be challenging for human experts. However, Early detection of Diabetic Retinopathy can significantly help to avoid permanent vision loss. The capability of computer-aided detection systems to accurately and efficiently detect the diabetic retino…
Strategy for Rapid Diabetic Retinopathy Exposure Based on Enhanced Feature Extraction Processing
M. Kumarasamy College of Engineering（M. Kumarasamy工程学院）；Saveetha Institute of Medical and Technical Sciences（Saveetha医学与技术学院）；Saveetha School of Engineering（Saveetha工程学院）
Deep learning-based models are developed to automatically detect if a retina image is `referable' in diabetic retinopathy (DR) screening. However, their classification accuracy degrades as the input images distributionally shift from their training distribution. Further, even if the input is not a retina image, a standard DR classifier produces a high confident prediction that the image is `referable'. Our paper presents a Dirichlet Prior Network-based framework to address this issue. It utilizes an out-of-distribution (OOD) detector model and a DR classification model to improve generalizabil…
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Singapore Eye Research Institute（新加坡眼科研究所）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Institute for Molecular and Clinical Ophthalmology（分子与临床眼科研究所）；National University of Singapore（新加坡国立大学）
In this paper, we propose an explainable and interpretable diabetic retinopathy (ExplainDR) classification model based on neural-symbolic learning. To gain explainability, a highlevel symbolic representation should be considered in decision making. Specifically, we introduce a human-readable symbolic representation, which follows a taxonomy style of diabetic retinopathy characteristics related to eye health conditions to achieve explainability. We then include humanreadable features obtained from the symbolic representation in the disease prediction. Experimental results on a diabetic retinopa…
Research and Technology Development Center（研究与技术开发中心）；Sharda University（夏尔达大学）
Early diagnosis of diabetic retinopathy for treatment of the disease has been failing to reach diabetic people living in rural areas. Shortage of trained ophthalmologists, limited availability of healthcare centers, and expensiveness of diagnostic equipment are among the reasons. Although many deep learning-based automatic diagnosis of diabetic retinopathy techniques have been implemented in the literature, these methods still fail to provide a point-of-care diagnosis. This raises the need for an independent diagnostic of diabetic retinopathy that can be used by a non-expert. Recently the usag…
