High fidelity segmentation of both macro and microvascular structure of the retina plays a pivotal role in determining degenerative retinal diseases, yet it is a difficult problem. Due to successive resolution loss in the encoding phase combined with the inability to recover this lost information in the decoding phase, autoencoding based segmentation approaches are limited in their ability to extract retinal microvascular structure. We propose RV-GAN, a new multi-scale generative architecture for accurate retinal vessel segmentation to alleviate this. The proposed architecture uses two generators and two multi-scale autoencoding discriminators for better microvessel localization and segmentation. In order to avoid the loss of fidelity suffered by traditional GAN-based segmentation systems, we introduce a novel weighted feature matching loss. This new loss incorporates and prioritizes features from the discriminator's decoder over the encoder. Doing so combined with the fact that the discriminator's decoder attempts to determine real or fake images at the pixel level better preserves macro and microvascular structure. By combining reconstruction and weighted feature matching loss, the proposed architecture achieves an area under the curve (AUC) of 0.9887, 0.9914, and 0.9887 in pixel-wise segmentation of retinal vasculature from three publicly available datasets, namely DRIVE, CHASE-DB1, and STARE, respectively. Additionally, RV-GAN outperforms other architectures in two additional relevant metrics, mean intersection-over-union (Mean-IOU) and structural similarity measure (SSIM).
University of Nevada, Reno（内华达大学里诺分校）；Deakin University（迪肯大学）；Houston Eye Associates（休斯顿眼科协会）；University of Nevada School of Medicine（内华达大学医学院）
Carrying out clinical diagnosis of retinal vascular degeneration using Fluorescein Angiography (FA) is a time consuming process and can pose significant adverse effects on the patient. Angiography requires insertion of a dye that may cause severe adverse effects and can even be fatal. Currently, there are no non-invasive systems capable of generating Fluorescein Angiography images. However, retinal fundus photography is a non-invasive imaging technique that can be completed in a few seconds. In order to eliminate the need for FA, we propose a conditional generative adversarial network (GAN) to…
Generation of Structurally Realistic Retinal Fundus Images with Diffusion Models
Seoul National University College of Medicine（首尔大学医学院）；Seoul National University Bundang Hospital（首尔大学盆唐医院）；VUNO, Inc.（VUNO公司）；Kookmin University（国民大学）
University of Nevada, Reno（内华达大学雷诺分校）；Deakin University（迪肯大学）；Houston Eye Associates（休斯顿眼科协会）
Fluorescein Angiography (FA) is a technique that employs the designated camera for Fundus photography incorporating excitation and barrier filters. FA also requires fluorescein dye that is injected intravenously, which might cause adverse effects ranging from nausea, vomiting to even fatal anaphylaxis. Currently, no other fast and non-invasive technique exists that can generate FA without coupling with Fundus photography. To eradicate the need for an invasive FA extraction procedure, we introduce an Attention-based Generative network that can synthesize Fluorescein Angiography from Fundus imag…
Fundus2Video: Cross-Modal Angiography Video Generation from Static Fundus Photography with Clinical Knowledge Guidance
The Hong Kong Polytechnic University（香港理工大学）；Clemson University（克莱姆森大学）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Monash University（莫纳什大学）；Sun Yat-sen University（中山大学）
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences（中国科学院苏州生物医学工程技术研究所）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）；the Third People’s Hospital of Changzhou（常州市第三人民医院）
Fluorescein angiography can provide a map of retinal vascular structure and function, which is commonly used in ophthalmology diagnosis, however, this imaging modality may pose risks of harm to the patients. To help physicians reduce the potential risks of diagnosis, an image translation method is adopted. In this work, we proposed a conditional generative adversarial network(GAN) - based method to directly learn the mapping relationship between structure fundus images and fundus fluorescence angiography images. Moreover, local saliency maps, which define each pixel's importance, are used to d…
University of Nevada, Reno（内华达大学雷诺分校）；Houston Eye Associates（休斯顿眼科协会）
In Fluorescein Angiography (FA), an exogenous dye is injected in the bloodstream to image the vascular structure of the retina. The injected dye can cause adverse reactions such as nausea, vomiting, anaphylactic shock, and even death. In contrast, color fundus imaging is a non-invasive technique used for photographing the retina but does not have sufficient fidelity for capturing its vascular structure. The only non-invasive method for capturing retinal vasculature is optical coherence tomography-angiography (OCTA). However, OCTA equipment is quite expensive, and stable imaging is limited to s…
Accurate measurements of abnormalities like Artery-Vein ratio and tortuosity in fundus images is an actively researched task. Most of the research seems to compute such features independently. However, in this work, we have devised a fully automated technique to measure any vascular abnormalities. This paper is a follow-up paper on vessel topology estimation and extraction, we use the extracted topology to perform A-V state-of-the-art Artery-Vein classification, AV ratio calculation, and vessel tortuosity measurement, all fully automated. Existing techniques tend to only work on the partial re…
Fundus to Fluorescein Angiography Video Generation as a Retinal Generative Foundation Model
The Hong Kong Polytechnic University（香港理工大学）；Swiss Federal Institute of Technology Lausanne (EPFL)（瑞士洛桑联邦理工学院）；Clemson University（克莱姆森大学）；University of Science and Technology of China（中国科学技术大学）；Research Centre for SHARP Vision (RSCV), The Hong Kong Polytechnic University（香港理工大学SHARP视觉研究中心）；Centre for Eye and Vision Research (CEVR)（眼科视觉研究中心）
Fundus2Globe: Generative AI-Driven 3D Digital Twins for Personalized Myopia Management
School of Optometry, The Hong Kong Polytechnic University（香港理工大学验光学院）；Swiss Federal Institute of Technology Lausanne (EPFL)（瑞士洛桑联邦理工学院）；Centre for Eye and Vision Research (CEVR)（眼与视觉研究中心）
UWAFA-GAN: Ultra-Wide-Angle Fluorescein Angiography Transformation via Multi-scale Generation and Registration Enhancement
Hangzhou Dianzi University（杭州电子科技大学）；Southern University of Science and Technology（南方科技大学）；Shenzhen University（深圳大学）；Misr Higher Institute for Commerce and Computers（米斯尔高等商务与计算机学院）
SpecF2M: A Spectral-Aware Multi-task Network Estimating Axial Length and Refractive Error from Pediatric Fundus Photographs
The Chinese University of Hong Kong（香港中文大学）；Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Capital Medical University（首都医科大学）；Zhengzhou Second Hospital（郑州市第二人民医院）
PALM: Open Fundus Photograph Dataset with Pathologic Myopia Recognition and Anatomical Structure Annotation
Baidu Inc.（百度公司）；Sun Yat-sen University（中山大学）；Agency for Science, Technology and Research（新加坡科技研究局）；CONICET（阿根廷国家科学与技术研究理事会）；UNICEN（阿根廷国立中央大学）；Medical University of Vienna（维也纳医科大学）
National Eye Institute (NEI), National Institutes of Health (NIH)（美国国立卫生研究院国立眼科研究所）；National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH)（美国国立卫生研究院国家医学图书馆国家生物技术信息中心）
Purpose: To assess the utility of deep learning in the detection of geographic atrophy (GA) from color fundus photographs; secondary aim to explore potential utility in detecting central GA (CGA). Design: A deep learning model was developed to detect the presence of GA in color fundus photographs, and two additional models to detect CGA in different scenarios. Participants: 59,812 color fundus photographs from longitudinal follow up of 4,582 participants in the AREDS dataset. Gold standard labels were from human expert reading center graders using a standardized protocol. Methods: A deep learn…
Do Not Break the Vessels: Structure-Preserving Mean Flow for Vascular Image Translation
School of Biological Science and Medical Engineering, Southeast University（东南大学生物科学与医学工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）；Zhejiang University（浙江大学）；Suzhou Microclear Medical Instruments Co.（苏州微清医疗器械有限公司）
Helmholtz Zentrum München（慕尼黑亥姆霍兹中心）；Technical University of Munich（慕尼黑工业大学）
Fundus photography is the primary method for retinal imaging and essential for diabetic retinopathy prevention. Automated segmentation of fundus photographs would improve the quality, capacity, and cost-effectiveness of eye care screening programs. However, current segmentation methods are not robust towards the diversity in imaging conditions and pathologies typical for real-world clinical applications. To overcome these limitations, we utilized contrastive self-supervised learning to exploit the large variety of unlabeled fundus images in the publicly available EyePACS dataset. We pre-traine…
In this paper, we propose two distinct solutions to the problem of Diabetic Retinopathy (DR) classification. In the first approach, we introduce a shallow neural network architecture. This model performs well on classification of the most frequent classes while fails at classifying the less frequent ones. In the second approach, we use transfer learning to re-train the last modified layer of a very deep neural network to improve the generalization ability of the model to the less frequent classes. Our results demonstrate superior abilities of transfer learning in DR classification of less freq…
Arizona State University（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Mayo Clinic（梅奥诊所）
Real-world non-mydriatic retinal fundus photography is prone to artifacts, imperfections and low-quality when certain ocular or systemic co-morbidities exist. Artifacts may result in inaccuracy or ambiguity in clinical diagnoses. In this paper, we proposed a simple but effective end-to-end framework for enhancing poor-quality retinal fundus images. Leveraging the optimal transport theory, we proposed an unpaired image-to-image translation scheme for transporting low-quality images to their high-quality counterparts. We theoretically proved that a Generative Adversarial Networks (GAN) model wit…
Centro de Investigación CITIC（CITIC研究中心）；Universidade da Coruña（拉科鲁尼亚大学）；Instituto de Investigación Biomédica de A Coruña (INIBIC)（拉科鲁尼亚生物医学研究所（INIBIC））
The study of the retinal vasculature is a fundamental stage in the screening and diagnosis of many diseases. A complete retinal vascular analysis requires to segment and classify the blood vessels of the retina into arteries and veins (A/V). Early automatic methods approached these segmentation and classification tasks in two sequential stages. However, currently, these tasks are approached as a joint semantic segmentation task, as the classification results highly depend on the effectiveness of the vessel segmentation. In that regard, we propose a novel approach for the simultaneous segmentat…
Peking University（北京大学）；Peking University First Hospital（北京大学第一医院）；MGH/BWH Center for Clinical Data Science（麻省总医院/布莱根妇女医院临床数据科学中心）
Segmenting coronary arteries is challenging, as classic unsupervised methods fail to produce satisfactory results and modern supervised learning (deep learning) requires manual annotation which is often time-consuming and can some time be infeasible. To solve this problem, we propose a knowledge transfer based shape-consistent generative adversarial network (SC-GAN), which is an annotation-free approach that uses the knowledge from publicly available annotated fundus dataset to segment coronary arteries. The proposed network is trained in an end-to-end fashion, generating and segmenting synthe…
MAF-Net: Multiple attention-guided fusion network for fundus vascular image segmentation
School of Electrical and Automation Engineering, East China Jiaotong University（华东交通大学电气与自动化工程学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）
HyMNet: a Multimodal Deep Learning System for Hypertension Classification using Fundus Photographs and Cardiometabolic Risk Factors
King Abdullah International Medical Research Center（阿卜杜拉国王国际医学研究中心）；King Saud bin Abdulaziz University for Health Sciences（沙特·本·阿卜杜勒阿齐兹国王健康科学大学）；King Abdulaziz Medical City, Ministry of National Guard Health Affairs（国民警卫队卫生事务部阿卜杜勒阿齐兹国王医疗城）；Prince Mohammad bin Abdulaziz Hospital, Ministry of National Guard Health Affairs（国民警卫队卫生事务部穆罕默德·本·阿卜杜勒阿齐兹亲王医院）
Fundus photography (FP) remains the primary imaging modality in screening various retinal diseases including age-related macular degeneration, diabetic retinopathy and glaucoma. FP allows the clinician to examine the ocular fundus structures such as the macula, the optic disc (OD) and retinal vessels, whose visibility and clarity in an FP image remain central to ensuring diagnostic accuracy, and hence determine the diagnostic quality (DQ). Images with low DQ, resulting from eye movement, improper illumination and other possible causes, should obviously be recaptured. However, the technician, o…
One of the leading causes of blindness is glaucoma, which is challenging to detect since it remains asymptomatic until the symptoms are severe. Thus, diagnosis is usually possible until the markers are easy to identify, i.e., the damage has already occurred. Early identification of glaucoma is generally made based on functional, structural, and clinical assessments. However, due to the nature of the disease, researchers still debate which markers qualify as a consistent glaucoma metric. Deep learning methods have partially solved this dilemma by bypassing the marker identification stage and an…
Cross-Fundus Transformer for Multi-modal Diabetic Retinopathy Grading with Cataract
Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Eye Diseases Prevention & Treatment Center（上海眼病防治中心）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）
Southern University of Science and Technology（南方科技大学）；A*STAR（新加坡科技研究局）；New York University（纽约大学）；Chinese Academy of Sciences（中国科学院）；Shenzhen People’s Hospital（深圳市人民医院）
Fundus photography is a routine examination in clinics to diagnose and monitor ocular diseases. However, for cataract patients, the fundus image always suffers quality degradation caused by the clouding lens. The degradation prevents reliable diagnosis by ophthalmologists or computer-aided systems. To improve the certainty in clinical diagnosis, restoration algorithms have been proposed to enhance the quality of fundus images. Unfortunately, challenges remain in the deployment of these algorithms, such as collecting sufficient training data and preserving retinal structures. In this paper, to …
Pulsatile changes in retinal vascular geometry over the cardiac cycle have clinical implications for the diagnosis of ocular and systemic vascular diseases, including ischemia, coronary heart diseases, and diabetes mellitus and its complications. Thus, analysis of the pulsatile changes over the cardiac cycle is a potential non-invasive assessment for the presence of ocular and systemic vascular diseases. The cardiac rhythm influences these pulsatile changes in the retina, is a result of the change in blood volumetric flow entering the ophthalmic-vascular system under a certain level of intraoc…
Fundus-Enhanced Disease-Aware Distillation Model for Retinal Disease Classification from OCT Images
The Hong Kong University of Science and Technology（香港科技大学）；Guangdong Provincial Hospital of Integrated Traditional Chinese and Western Medicine（广东省中西医结合医院）；Guangdong Weiren Meditech Co., Ltd（广东唯仁医疗科技有限公司）
Massachusetts General Hospital（麻省总医院）；Casey Eye Institute（凯西眼科研究所）；Brigham and Women’s Hospital（布莱根妇女医院）
Generative adversarial networks (GANs) are a class of unsupervised machine learning algorithms that can produce realistic images from randomly-sampled vectors in a multi-dimensional space. Until recently, it was not possible to generate realistic high-resolution images using GANs, which has limited their applicability to medical images that contain biomarkers only detectable at native resolution. Progressive growing of GANs is an approach wherein an image generator is trained to initially synthesize low resolution synthetic images (8x8 pixels), which are then fed to a discriminator that distin…
University of California, Berkeley（加利福尼亚大学伯克利分校）；International Computer Science Institute（国际计算机科学研究所）；University of California, Davis（加利福尼亚大学戴维斯分校）
Many diseases are classified based on human-defined rubrics that are prone to bias. Supervised neural networks can automate the grading of retinal fundus images, but require labor-intensive annotations and are restricted to the specific trained task. Here, we employed an unsupervised network with Non-Parametric Instance Discrimination (NPID) to grade age-related macular degeneration (AMD) severity using fundus photographs from the Age-Related Eye Disease Study (AREDS). Our unsupervised algorithm demonstrated versatility across different AMD classification schemes without retraining, and achiev…
Integrating Deep Learning with Fundus and Optical Coherence Tomography for Cardiovascular Disease Prediction
University of Leeds（利兹大学）；University of Manchester（曼彻斯特大学）；NIHR Manchester Biomedical Research Centre（NIHR曼彻斯特生物医学研究中心）；Manchester Academic Health Science Centre（曼彻斯特学术健康科学中心）；Christabel Pankhurst Institute（克里斯塔贝尔·潘克赫斯特研究所）
The detection of retinal blood vessels, especially the changes of small vessel condition is the most important indicator to identify the vascular network of the human body. Existing techniques focused mainly on shape of the large vessels, which is not appropriate for the disconnected small and isolated vessels. Paying attention to the low contrast small blood vessel in fundus region, first time we proposed to combine graph based smoothing regularizer with the loss function in the U-net framework. The proposed regularizer treated the image as two graphs by calculating the graph laplacians on ve…
A Generic Fundus Image Enhancement Network Boosted by Frequency Self-supervised Representation Learning
Southern University of Science and Technology（南方科技大学）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；New York University（纽约大学）；Beijing Information Science and Technology University（北京信息科技大学）
Sun Yat-sen University（中山大学）；Baidu Inc.（百度公司）；Agency for Science, Technology and Research（新加坡科技研究局）；CONICET（阿根廷国家科学技术研究委员会）；UNICEN（阿根廷中央国立大学）；Medical University of Vienna（维也纳医科大学）；VUNO Inc.（VUNO公司）；Tencent（腾讯）
With the rapid development of artificial intelligence (AI) in medical image processing, deep learning in color fundus photography (CFP) analysis is also evolving. Although there are some open-source, labeled datasets of CFPs in the ophthalmology community, large-scale datasets for screening only have labels of disease categories, and datasets with annotations of fundus structures are usually small in size. In addition, labeling standards are not uniform across datasets, and there is no clear information on the acquisition device. Here we release a multi-annotation, multi-quality, and multi-dev…
VARPA Research Group, Instituto de Investigación Biomédica de A Coruña (INIBIC), Universidade da Coruña（拉科鲁尼亚大学生物医学研究所（INIBIC）VARPA研究组）；Centro de Investigación CITIC, Universidade da Coruña（拉科鲁尼亚大学CITIC研究中心）
In ophthalmology, the study of the retinal microcirculation is a key issue in the analysis of many ocular and systemic diseases, like hypertension or diabetes. This motivates the research on improving the retinal vasculature segmentation. Nowadays, Fully Convolutional Neural Networks (FCNs) usually represent the most successful approach to image segmentation. However, the success of these models is conditioned by an adequate selection and adaptation of the architectures and techniques used, as well as the availability of a large amount of annotated data. These two issues become specially relev…
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）
Fundus photography has routinely been used to document the presence and severity of retinal degenerative diseases such as age-related macular degeneration (AMD), glaucoma, and diabetic retinopathy (DR) in clinical practice, for which the fovea and optic disc (OD) are important retinal landmarks. However, the occurrence of lesions, drusen, and other retinal abnormalities during retinal degeneration severely complicates automatic landmark detection and segmentation. Here we propose HBA-U-Net: a U-Net backbone enriched with hierarchical bottleneck attention. The network consists of a novel bottle…
Eötvös Loránd University（罗兰大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Jiangxi Normal University（江西师范大学）；Chinese Academy of Science（中国科学院）；Qinghai University（青海大学）
Reliable segmentation of retinal vessels can be employed as a way of monitoring and diagnosing certain diseases, such as diabetes and hypertension, as they affect the retinal vascular structure. In this work, we propose the Residual Spatial Attention Network (RSAN) for retinal vessel segmentation. RSAN employs a modified residual block structure that integrates DropBlock, which can not only be utilized to construct deep networks to extract more complex vascular features, but can also effectively alleviate the overfitting. Moreover, in order to further improve the representation capability of t…
Broad Institute of MIT and Harvard（麻省理工学院-哈佛大学博德研究所）；Korea University（高丽大学）；NAVER CLOVA；Harvard Medical School（哈佛医学院）；Ophthalmic Consultants of Boston（波士顿眼科顾问诊所）
The application of modern machine learning to retinal image analyses offers valuable insights into a broad range of human health conditions beyond ophthalmic diseases. Additionally, data sharing is key to fully realizing the potential of machine learning models by providing a rich and diverse collection of training data. However, the personally-identifying nature of retinal images, encompassing the unique vascular structure of each individual, often prevents this data from being shared openly. While prior works have explored image de-identification strategies based on synthetic averaging of im…
Early detection of microaneurysms (MAs), the first sign of Diabetic Retinopathy (DR), is an essential first step in automated detection of DR to prevent vision loss and blindness. This study presents a novel and different algorithm for automatic detection of MAs in fluorescein angiography (FA) fundus images, based on Radon transform (RT) and multi-overlapping windows. This project addresses a novel method, in detection of retinal land marks and lesions to diagnose the DR. At the first step, optic nerve head (ONH) was detected and masked. In preprocessing stage, top-hat transformation and avera…
We present a fully automatic, graph-based technique for extracting the retinal vascular topology -- that is, how different vessels are connected to each other -- given a single color fundus image. Determining this connectivity is very challenging because vessels cross each other in a 2D image, obscuring their true paths. We quantitatively validated the usefulness of our extraction method by using it to achieve comparable state-of-the-art results in retinal artery-vein classification. Our proposed approach works as follows: We first segment the retinal vessels using our previously developed sta…
