Synthesizing New Retinal Symptom Images by Multiple Generative Models
National Taiwan University（台湾大学）；King Abdullah University of Science and Technology（沙特阿拉伯国王阿卜杜拉科技大学）；Georgia Institute of Technology（佐治亚理工学院）；Bern University Hospital（伯尔尼大学医院）；Biological and Environmental Sciences and Engineering Division, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology（沙特阿拉伯国王阿卜杜拉科技大学生物与环境科学与工程部门、计算机、电气与数学科学与工程部门）；Department of Ophthalmology, Bern University Hospital（伯尔尼大学医院眼科）
Age-Related Macular Degeneration (AMD) is an asymptomatic retinal disease which may result in loss of vision. There is limited access to high-quality relevant retinal images and poor understanding of the features defining sub-classes of this disease. Motivated by recent advances in machine learning we specifically explore the potential of generative modeling, using Generative Adversarial Networks (GANs) and style transferring, to facilitate clinical diagnosis and disease understanding by feature extraction. We design an analytic pipeline which first generates synthetic retinal images from clinical images; a subsequent verification step is applied. In the synthesizing step we merge GANs (DCGANs and WGANs architectures) and style transferring for the image generation, whereas the verified step controls the accuracy of the generated images. We find that the generated images contain sufficient pathological details to facilitate ophthalmologists' task of disease classification and in discovery of disease relevant features. In particular, our system predicts the drusen and geographic atrophy sub-classes of AMD. Furthermore, the performance using CFP images for GANs outperforms the classification based on using only the original clinical dataset. Our results are evaluated using existing classifier of retinal diseases and class activated maps, supporting the predictive power of the synthetic images and their utility for feature extraction. Our code examples are available online.
Beihang University（北京航空航天大学）；National Institute of Informatics（国立情报学研究所）；Peng Cheng Laboratory（鹏城实验室）；Central South University（中南大学）
Though deep learning has shown successful performance in classifying the label and severity stage of certain disease, most of them give few evidence on how to make prediction. Here, we propose to exploit the interpretability of deep learning application in medical diagnosis. Inspired by Koch's Postulates, a well-known strategy in medical research to identify the property of pathogen, we define a pathological descriptor that can be extracted from the activated neurons of a diabetic retinopathy detector. To visualize the symptom and feature encoded in this descriptor, we propose a GAN based meth…
University of Nevada, Reno（内华达大学雷诺分校）；Deakin University（迪肯大学）；Houston Eye Associates（休斯顿眼科协会）
Fluorescein Angiography (FA) is a technique that employs the designated camera for Fundus photography incorporating excitation and barrier filters. FA also requires fluorescein dye that is injected intravenously, which might cause adverse effects ranging from nausea, vomiting to even fatal anaphylaxis. Currently, no other fast and non-invasive technique exists that can generate FA without coupling with Fundus photography. To eradicate the need for an invasive FA extraction procedure, we introduce an Attention-based Generative network that can synthesize Fluorescein Angiography from Fundus imag…
Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）；RIKEN AIP（理化学研究所人工智能研究中心）；University of Tokyo（东京大学）；Ningbo Institute of Industrial Technology, CAS（中国科学院宁波工业技术研究院）
Though deep learning has shown successful performance in classifying the label and severity stage of certain diseases, most of them give few explanations on how to make predictions. Inspired by Koch's Postulates, the foundation in evidence-based medicine (EBM) to identify the pathogen, we propose to exploit the interpretability of deep learning application in medical diagnosis. By determining and isolating the neuron activation patterns on which diabetic retinopathy (DR) detector relies to make decisions, we demonstrate the direct relation between the isolated neuron activation and lesions for…
INESC TEC（系统与计算机工程研究所 - 科技与科学）；Stephen A Wynn Institute for Vision Research（Stephen A Wynn视觉研究所）；University of Iowa Hospital and Clinics（爱荷华大学医院和诊所）；IDx LLC（IDx有限责任公司）；University of Porto（波尔图大学）
Synthesizing images of the eye fundus is a challenging task that has been previously approached by formulating complex models of the anatomy of the eye. New images can then be generated by sampling a suitable parameter space. In this work, we propose a method that learns to synthesize eye fundus images directly from data. For that, we pair true eye fundus images with their respective vessel trees, by means of a vessel segmentation technique. These pairs are then used to learn a mapping from a binary vessel tree to a new retinal image. For this purpose, we use a recent image-to-image translatio…
Indian Institute of Information Technology, Sri City（印度信息技术学院，斯里城）
Image to Image Translation (I2I) is a challenging computer vision problem used in numerous domains for multiple tasks. Recently, ophthalmology became one of the major fields where the application of I2I is increasing rapidly. One such application is the generation of synthetic retinal optical coherence tomographic (OCT) scans. Existing I2I methods require training of multiple models to translate images from normal scans to a specific pathology: limiting the use of these models due to their complexity. To address this issue, we propose an unsupervised multi-domain I2I network with pre-trained s…
Enhancing Retinal Vessel Segmentation Generalization via Layout-Aware Generative Modelling
Faculty of Biomedical Engineering, Technion-IIT（以色列理工学院生物医学工程学院）；Faculty of Mathematics, Technion-IIT（以色列理工学院数学学院）；KU Leuven（鲁汶大学）；UZ Leuven（鲁汶大学医院）；São Paulo Federal University（圣保罗联邦大学）；Massachusetts Institute of Technology（麻省理工学院）；School of Electrical and Computer Engineering, Ben-Gurion University of the Negev（本-古里安大学内盖夫分校电气与计算机工程学院）
University of Nevada, Reno（内华达大学雷诺分校）；Houston Eye Associates（休斯顿眼科协会）
In Fluorescein Angiography (FA), an exogenous dye is injected in the bloodstream to image the vascular structure of the retina. The injected dye can cause adverse reactions such as nausea, vomiting, anaphylactic shock, and even death. In contrast, color fundus imaging is a non-invasive technique used for photographing the retina but does not have sufficient fidelity for capturing its vascular structure. The only non-invasive method for capturing retinal vasculature is optical coherence tomography-angiography (OCTA). However, OCTA equipment is quite expensive, and stable imaging is limited to s…
One of the leading causes of blindness is glaucoma, which is challenging to detect since it remains asymptomatic until the symptoms are severe. Thus, diagnosis is usually possible until the markers are easy to identify, i.e., the damage has already occurred. Early identification of glaucoma is generally made based on functional, structural, and clinical assessments. However, due to the nature of the disease, researchers still debate which markers qualify as a consistent glaucoma metric. Deep learning methods have partially solved this dilemma by bypassing the marker identification stage and an…
University of Siena（锡耶纳大学）；University of Florence（佛罗伦萨大学）
In recent years, the use of deep learning is becoming increasingly popular in computer vision. However, the effective training of deep architectures usually relies on huge sets of annotated data. This is critical in the medical field where it is difficult and expensive to obtain annotated images. In this paper, we use Generative Adversarial Networks (GANs) for synthesizing high quality retinal images, along with the corresponding semantic label-maps, to be used instead of real images during the training process. Differently from other previous proposals, we suggest a two step approach: first, …
Broad Institute of MIT and Harvard（麻省理工学院-哈佛大学博德研究所）；Korea University（高丽大学）；NAVER CLOVA；Harvard Medical School（哈佛医学院）；Ophthalmic Consultants of Boston（波士顿眼科顾问诊所）
The application of modern machine learning to retinal image analyses offers valuable insights into a broad range of human health conditions beyond ophthalmic diseases. Additionally, data sharing is key to fully realizing the potential of machine learning models by providing a rich and diverse collection of training data. However, the personally-identifying nature of retinal images, encompassing the unique vascular structure of each individual, often prevents this data from being shared openly. While prior works have explored image de-identification strategies based on synthetic averaging of im…
Beijing Institute of Technology（北京理工大学）；Bioinformatics Institute, A*STAR（A*STAR生物信息研究所）；National University of Singapore（新加坡国立大学）
This paper aims at synthesizing filamentary structured images such as retinal fundus images and neuronal images, as follows: Given a ground-truth, to generate multiple realistic looking phantoms. A ground-truth could be a binary segmentation map containing the filamentary structured morphology, while the synthesized output image is of the same size as the ground-truth and has similar visual appearance to what have been presented in the training set. Our approach is inspired by the recent progresses in generative adversarial nets (GANs) as well as image style transfer. In particular, it is dedi…
University of Chinese Academy of Sciences（中国科学院大学）；Inception Institute of Artificial Intelligence（启元人工智能研究院）；UBTech Research（优必选研究院）；ShanghaiTech University（上海科技大学）；Southern University of Science and Technology（南方科技大学）
Generative Adversarial Networks (GANs) have the capability of synthesizing images, which have been successfully applied to medical image synthesis tasks. However, most of existing methods merely consider the global contextual information and ignore the fine foreground structures, e.g., vessel, skeleton, which may contain diagnostic indicators for medical image analysis. Inspired by human painting procedure, which is composed of stroking and color rendering steps, we propose a Sketching-rendering Unconditional Generative Adversarial Network (SkrGAN) to introduce a sketch prior constraint to gui…
REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Prediction
J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida（佛罗里达大学J. Crayton Pruitt家族生物医学工程系）；Research Institute of Electrical Communication, Tohoku University（东北大学电气通信研究所）；Department of Applied Mathematics & Statistics, Stony Brook University（石溪大学应用数学与统计学系）；Department of Computer Science, Stony Brook University（石溪大学计算机科学系）
Propagating Structural Guidance: Synthesizing Fluorescein Angiography from Fundus Images and Sparse OCT Scans
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education（教育部新一代人工智能技术及其跨学科应用重点实验室）；Tianyuan Honors School, Nanjing Medical University（南京医科大学天元荣誉学院）；Nanjing University of Science and Technology（南京理工大学）；Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院眼科）
The College of New Jersey（新泽西学院）；University of Oxford（牛津大学）；Ludwig Institute for Cancer Research（路德维希癌症研究所）
Disease complications can alter vascular network morphology and disrupt tissue functioning. Diabetic retinopathy, for example, is a complication of types 1 and 2 diabetes mellitus that can cause blindness. Microvascular diseases are assessed by visual inspection of retinal images, but this can be challenging when diseases exhibit silent symptoms or patients cannot attend in-person meetings. We examine the performance of machine learning algorithms in detecting microvascular disease when trained on statistical and topological summaries of segmented retinal vascular images. We apply our methods …
Shahid Beheshti University（沙希德·贝赫什提大学）；Shahid Beheshti University of Medical Sciences（沙希德·贝赫什提医科大学）
Intra retinal fluids or Cysts are one of the important symptoms of macular pathologies that are efficiently visualized in OCT images. Automatic segmentation of these abnormalities has been widely investigated in medical image processing studies. In this paper, we propose a new U-Net-based approach for Intra retinal cyst segmentation across different vendors that improves some of the challenges faced by previous deep-based techniques. The proposed method has two main steps: 1- prior information embedding and input data adjustment, and 2- IRC segmentation model. In the first step, we inject the …
Strategy for Rapid Diabetic Retinopathy Exposure Based on Enhanced Feature Extraction Processing
M. Kumarasamy College of Engineering（M. Kumarasamy工程学院）；Saveetha Institute of Medical and Technical Sciences（Saveetha医学与技术学院）；Saveetha School of Engineering（Saveetha工程学院）
A Disease-Specific Foundation Model Using Over 100K Fundus Images: Release and Validation for Abnormality and Multi-Disease Classification on Downstream Tasks
Seoul National University（首尔大学）；Seoul National University Hospital（首尔大学附属医院）
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）；Capital Medical University（首都医科大学）；Deepwise AI Lab（深睿医疗AI实验室）；Beijing Stars Universal Technology Co Ltd（北京星辰万联科技有限公司）
Forecasting Parapapillary atrophy (PPA), i.e., a symptom related to most irreversible eye diseases, provides an alarm for implementing an intervention to slow down the disease progression at early stage. A key question for this forecast is: how to fully utilize the historical data (e.g., retinal image) up to the current stage for future disease prediction? In this paper, we provide an answer with a novel framework, namely \textbf{D}isease \textbf{F}orecast via \textbf{P}rogression \textbf{L}earning (\textbf{DFPL}), which exploits the irreversibility prior (i.e., cannot be reversed once diagnos…
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Glaucoma is one of the most severe eye diseases, characterized by rapid progression and leading to irreversible blindness. It is often the case that diagnostics is carried out when one's sight has already significantly degraded due to the lack of noticeable symptoms at early stage of the disease. Regular glaucoma screenings of the population shall improve early-stage detection, however the desirable frequency of etymological checkups is often not feasible due to the excessive load imposed by manual diagnostics on limited number of specialists. Considering the basic methodology to detect glauco…
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）
Diabetic retinopathy (DR) is the most common form of diabetic eye disease. Retinopathy can affect all diabetic patients and becomes particularly dangerous, increasing the risk of blindness, if it is left untreated. The success rate of its curability solemnly depends on diagnosis at an early stage. The development of automated computer aided disease diagnosis tools could help in faster detection of symptoms with a wider reach and reasonable cost. This paper proposes a method for the automated segmentation of retinal lesions and optic disk in fundus images using a deep fully convolutional neural…
CACTUS as a Reliable Tool for Early Classification of Age-related Macular Degeneration
Personal Health Data Science Team, Sano Centre for Computational Personalised Medicine（个人健康数据科学团队，Sano计算个性化医学中心）；The Wellcome-Wolfson Institute for Experimental Medicine, School of Medicine Dentistry and Biomedical Science, Queen's University Belfast（Wellcome-Wolfson实验医学研究所，医学牙科与生物医学科学学院，女王大学贝尔法斯特）；Institute of Clinical Sciences Building A, Queen’s University Belfast（临床科学研究所A楼，女王大学贝尔法斯特）；Department of Epidemiology, Erasmus Medical Center（流行病学系，埃拉斯谟医学中心）；Department of Ophthalmology, Erasmus Medical Center（眼科系，埃拉斯谟医学中心）；Department of Ophthalmology, Radboud University Medical Centre（眼科系，拉德堡德大学医学中心）；Institute of Molecular and Clinical Ophthalmology Basel（分子与临床眼科研究所巴塞尔）
