Multimodal Transfer Learning-based Approaches for Retinal Vascular Segmentation
VARPA Research Group, Instituto de Investigación Biomédica de A Coruña (INIBIC), Universidade da Coruña（拉科鲁尼亚大学生物医学研究所（INIBIC）VARPA研究组）；Centro de Investigación CITIC, Universidade da Coruña（拉科鲁尼亚大学CITIC研究中心）
José Morano, Álvaro S. Hervella, Noelia Barreira, Jorge Novo, José Rouco
发表信息：
Proceedings of the 24th European Conference on Artificial Intelligence (ECAI 2020)
In ophthalmology, the study of the retinal microcirculation is a key issue in the analysis of many ocular and systemic diseases, like hypertension or diabetes. This motivates the research on improving the retinal vasculature segmentation. Nowadays, Fully Convolutional Neural Networks (FCNs) usually represent the most successful approach to image segmentation. However, the success of these models is conditioned by an adequate selection and adaptation of the architectures and techniques used, as well as the availability of a large amount of annotated data. These two issues become specially relevant when applying FCNs to medical image segmentation as, first, the existent models are usually adjusted from broad domain applications over photographic images, and second, the amount of annotated data is usually scarcer. In this work, we present multimodal transfer learning-based approaches for retinal vascular segmentation, performing a comparative study of recent FCN architectures. In particular, to overcome the annotated data scarcity, we propose the novel application of self-supervised network pretraining that takes advantage of existent unlabelled multimodal data. The results demonstrate that the self-supervised pretrained networks obtain significantly better vascular masks with less training in the target task, independently of the network architecture, and that some FCN architecture advances motivated for broad domain applications do not translate into significant improvements over the vasculature segmentation task.
Centro de Investigación CITIC（CITIC研究中心）；Universidade da Coruña（拉科鲁尼亚大学）；Instituto de Investigación Biomédica de A Coruña (INIBIC)（拉科鲁尼亚生物医学研究所（INIBIC））
The study of the retinal vasculature is a fundamental stage in the screening and diagnosis of many diseases. A complete retinal vascular analysis requires to segment and classify the blood vessels of the retina into arteries and veins (A/V). Early automatic methods approached these segmentation and classification tasks in two sequential stages. However, currently, these tasks are approached as a joint semantic segmentation task, as the classification results highly depend on the effectiveness of the vessel segmentation. In that regard, we propose a novel approach for the simultaneous segmentat…
Technion-IIT（以色列理工学院）；KU Leuven（鲁汶大学）；University Hospitals UZ Leuven（鲁汶大学附属医院）
Introduction: Blood vessels can be non-invasively visualized from a digital fundus image (DFI). Several studies have shown an association between cardiovascular risk and vascular features obtained from DFI. Recent advances in computer vision and image segmentation enable automatising DFI blood vessel segmentation. There is a need for a resource that can automatically compute digital vasculature biomarkers (VBM) from these segmented DFI. Methods: In this paper, we introduce a Python Vasculature BioMarker toolbox, denoted PVBM. A total of 11 VBMs were implemented. In particular, we introduce new…
Retinal vessel segmentation is of great interest for diagnosis of retinal vascular diseases. To further improve the performance of vessel segmentation, we propose IterNet, a new model based on UNet, with the ability to find obscured details of the vessel from the segmented vessel image itself, rather than the raw input image. IterNet consists of multiple iterations of a mini-UNet, which can be 4$\times$ deeper than the common UNet. IterNet also adopts the weight-sharing and skip-connection features to facilitate training; therefore, even with such a large architecture, IterNet can still learn …
Technical University of Munich（慕尼黑工业大学）；Imperial College London（帝国理工学院）；University of Zurich（苏黎世大学）
Optical coherence tomography angiography (OCTA) can non-invasively image the eye's circulatory system. In order to reliably characterize the retinal vasculature, there is a need to automatically extract quantitative metrics from these images. The calculation of such biomarkers requires a precise semantic segmentation of the blood vessels. However, deep-learning-based methods for segmentation mostly rely on supervised training with voxel-level annotations, which are costly to obtain. In this work, we present a pipeline to synthesize large amounts of realistic OCTA images with intrinsically matc…
MAF-Net: Multiple attention-guided fusion network for fundus vascular image segmentation
School of Electrical and Automation Engineering, East China Jiaotong University（华东交通大学电气与自动化工程学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）
Purpose: To demonstrate that retinal microvasculature per se is a reliable biomarker for Diabetic Retinopathy (DR) and, by extension, cardiovascular diseases. Methods: Deep Learning Convolutional Neural Networks (CNN) applied to color fundus images for semantic segmentation of the blood vessels and severity classification on both vascular and full images. Vessel reconstruction through harmonic descriptors is also used as a smoothing and de-noising tool. The mathematical background of the theory is also outlined. Results: For diabetic patients, at least 93.8% of DR No-Refer vs. Refer classifica…
Accurate measurements of abnormalities like Artery-Vein ratio and tortuosity in fundus images is an actively researched task. Most of the research seems to compute such features independently. However, in this work, we have devised a fully automated technique to measure any vascular abnormalities. This paper is a follow-up paper on vessel topology estimation and extraction, we use the extracted topology to perform A-V state-of-the-art Artery-Vein classification, AV ratio calculation, and vessel tortuosity measurement, all fully automated. Existing techniques tend to only work on the partial re…
Zhejiang University（浙江大学）；University of Alberta（阿尔伯塔大学）
Topological and geometrical analysis of retinal blood vessel is a cost-effective way for early detection of many common diseases. Meanwhile, automated vessel segmentation and vascular tree analysis are still lacking in terms of generalization capability. In this work, we construct a novel benchmark RETA with 81 labeled vessel masks aiming to facilitate retinal vessel analysis. A semi-automated coarse-to-fine workflow is proposed to annotating vessel pixels. During dataset construction, we strived to control inter-annotator variability and intra-annotator variability by performing multi-stage a…
University College London（伦敦大学学院）；UCL（伦敦大学学院）；NIHR Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology（英国国家健康研究所摩尔菲尔德眼科医院NHS基金会信托与伦敦大学学院眼科研究所生物医学研究中心）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences（惠康/英国工程与物理科学研究委员会介入与外科科学中心）
Estimating clinically-relevant vascular features following vessel segmentation is a standard pipeline for retinal vessel analysis, which provides potential ocular biomarkers for both ophthalmic disease and systemic disease. In this work, we integrate these clinical features into a novel vascular feature optimised loss function (VAFO-Loss), in order to regularise networks to produce segmentation maps, with which more accurate vascular features can be derived. Two common vascular features, vessel density and fractal dimension, are identified to be sensitive to intra-segment misclassification, wh…
National Taiwan University（台湾大学）；Chang Gung Memorial Hospital（长庚纪念医院）；Chang Gung University（长庚大学）
Automated vascular segmentation on optical coherence tomography angiography (OCTA) is important for the quantitative analyses of retinal microvasculature in neuroretinal and systemic diseases. Despite recent improvements, artifacts continue to pose challenges in segmentation. Our study focused on removing the speckle noise artifact from OCTA images when performing segmentation. Speckle noise is common in OCTA and is particularly prominent over large non-perfusion areas. It may interfere with the proper assessment of retinal vasculature. In this study, we proposed a novel Supervision Vessel Seg…
Systemic diseases such as diabetes, hypertension, atherosclerosis are among the leading causes of annual human mortality rate. It is suggested that retinal and conjunctival vascular tortuosity is a potential biomarker for such systemic diseases. Most importantly, it is observed that the tortuosity depends on the thickness of these vessels. Therefore, selective calculation of tortuosity within specific vessel thicknesses is required depending on the disease being analysed. In this paper, we propose a thickness sensitive vessel extraction framework that is primarily applicable for studies relate…
University of Nevada, Reno（内华达大学里诺分校）；Houston Eye Associates（休斯顿眼科协会）
High fidelity segmentation of both macro and microvascular structure of the retina plays a pivotal role in determining degenerative retinal diseases, yet it is a difficult problem. Due to successive resolution loss in the encoding phase combined with the inability to recover this lost information in the decoding phase, autoencoding based segmentation approaches are limited in their ability to extract retinal microvascular structure. We propose RV-GAN, a new multi-scale generative architecture for accurate retinal vessel segmentation to alleviate this. The proposed architecture uses two generat…
Generation of Structurally Realistic Retinal Fundus Images with Diffusion Models
Seoul National University College of Medicine（首尔大学医学院）；Seoul National University Bundang Hospital（首尔大学盆唐医院）；VUNO, Inc.（VUNO公司）；Kookmin University（国民大学）
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Jiangxi Normal University（江西师范大学）；Eötvös Loránd University（厄特沃什·罗兰大学）；Shenyang Aerospace University（沈阳航空航天大学）
Retinal vessel segmentation plays an imaportant role in the field of retinal image analysis because changes in retinal vascular structure can aid in the diagnosis of diseases such as hypertension and diabetes. In recent research, numerous successful segmentation methods for fundus images have been proposed. But for other retinal imaging modalities, more research is needed to explore vascular extraction. In this work, we propose an efficient method to segment blood vessels in Scanning Laser Ophthalmoscopy (SLO) retinal images. Inspired by U-Net, "feature map reuse" and residual learning, we pro…
Eötvös Loránd University（罗兰大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Jiangxi Normal University（江西师范大学）；Chinese Academy of Science（中国科学院）；Qinghai University（青海大学）
Reliable segmentation of retinal vessels can be employed as a way of monitoring and diagnosing certain diseases, such as diabetes and hypertension, as they affect the retinal vascular structure. In this work, we propose the Residual Spatial Attention Network (RSAN) for retinal vessel segmentation. RSAN employs a modified residual block structure that integrates DropBlock, which can not only be utilized to construct deep networks to extract more complex vascular features, but can also effectively alleviate the overfitting. Moreover, in order to further improve the representation capability of t…
Pulsatile changes in retinal vascular geometry over the cardiac cycle have clinical implications for the diagnosis of ocular and systemic vascular diseases, including ischemia, coronary heart diseases, and diabetes mellitus and its complications. Thus, analysis of the pulsatile changes over the cardiac cycle is a potential non-invasive assessment for the presence of ocular and systemic vascular diseases. The cardiac rhythm influences these pulsatile changes in the retina, is a result of the change in blood volumetric flow entering the ophthalmic-vascular system under a certain level of intraoc…
A Novel Hybrid Approach for Retinal Vessel Segmentation with Dynamic Long-Range Dependency and Multi-Scale Retinal Edge Fusion Enhancement
School of Artificial Intelligence（人工智能学院）；Hebei University of Technology（河北工业大学）；NHC Key Lab of Hormones and Development and Tianjin Key Lab of Metabolic Diseases（国家卫生健康委员会激素与发育重点实验室和天津代谢疾病重点实验室）；Tianjin Medical University Chu Hsien-I Memorial Hospital & Institute of Endocrinology（天津医科大学楚生一纪念医院及内分泌研究所）
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Hospital of the Peoples Liberation Army Joint Logistics Support Force（中国人民解放军联勤保障部队医院）；Jiangxi Normal University（江西师范大学）；Shenyang Institute of Computing Technology, Chinese Academy of Science（中国科学院沈阳计算技术研究所）
Retinal vessel segmentation is a vital step for the diagnosis of many early eye-related diseases. In this work, we propose a new deep learning model, namely Channel Attention Residual U-Net (CAR-UNet), to accurately segment retinal vascular and non-vascular pixels. In this model, we introduced a novel Modified Efficient Channel Attention (MECA) to enhance the discriminative ability of the network by considering the interdependence between feature maps. On the one hand, we apply MECA to the "skip connections" in the traditional U-shaped networks, instead of simply copying the feature maps of th…
This study is to characterize reflectance profiles of retinal blood vessels in optical coherence tomography (OCT), and to validate these vascular features to guide artery-vein classification in OCT angiography (OCTA) of human retina. Depth-resolved OCT reveals unique features of retinal arteries and veins. Retinal arteries show hyper-reflective boundaries at both upper (inner side towards the vitreous) and lower (outer side towards the choroid) walls. In contrary, retinal veins reveal hyper-reflectivity at the upper boundary only. Uniform lumen intensity was observed in both small and large ar…
University of Nevada, Reno（内华达大学雷诺分校）；Houston Eye Associates（休斯顿眼科协会）
In Fluorescein Angiography (FA), an exogenous dye is injected in the bloodstream to image the vascular structure of the retina. The injected dye can cause adverse reactions such as nausea, vomiting, anaphylactic shock, and even death. In contrast, color fundus imaging is a non-invasive technique used for photographing the retina but does not have sufficient fidelity for capturing its vascular structure. The only non-invasive method for capturing retinal vasculature is optical coherence tomography-angiography (OCTA). However, OCTA equipment is quite expensive, and stable imaging is limited to s…
Cixi Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences（中国科学院宁波材料技术与工程研究所慈溪生物医学工程研究所）；the Second Affiliated Hospital of Zhejiang University（浙江大学医学院附属第二医院）；Ningbo First Hospital（宁波市第一医院）；Edge Hill University（边山大学）；Ningbo University of Technology（宁波工程学院）
Automated detection of retinal structures, such as retinal vessels (RV), the foveal avascular zone (FAZ), and retinal vascular junctions (RVJ), are of great importance for understanding diseases of the eye and clinical decision-making. In this paper, we propose a novel Voting-based Adaptive Feature Fusion multi-task network (VAFF-Net) for joint segmentation, detection, and classification of RV, FAZ, and RVJ in optical coherence tomography angiography (OCTA). A task-specific voting gate module is proposed to adaptively extract and fuse different features for specific tasks at two levels: featur…
Automated segmentation of vascular map in retinal images endeavors a potential benefit in diagnostic procedure of different ocular diseases. In this paper, we suggest a new unsupervised retinal blood vessel segmentation approach using top-hat transformation, contrast-limited adaptive histogram equalization (CLAHE), and 2-D Gabor wavelet filters. Initially, retinal image is preprocessed using top-hat morphological transformation followed by CLAHE to enhance only the blood vessel pixels in the presence of exudates, optic disc, and fovea. Then, multiscale 2-D Gabor wavelet filters are applied on …
We present a fully automatic, graph-based technique for extracting the retinal vascular topology -- that is, how different vessels are connected to each other -- given a single color fundus image. Determining this connectivity is very challenging because vessels cross each other in a 2D image, obscuring their true paths. We quantitatively validated the usefulness of our extraction method by using it to achieve comparable state-of-the-art results in retinal artery-vein classification. Our proposed approach works as follows: We first segment the retinal vessels using our previously developed sta…
Simon Fraser University（西蒙弗雷泽大学）；University of British Columbia（不列颠哥伦比亚大学）；Medical University of Vienna（维也纳医科大学）；University of Zagreb（萨格勒布大学）
Purpose: Optical Coherence Tomography Angiography (OCT-A) permits visualization of the changes to the retinal circulation due to diabetic retinopathy (DR), a microvascular complication of diabetes. We demonstrate accurate segmentation of the vascular morphology for the superficial capillary plexus and deep vascular complex (SCP and DVC) using a convolutional neural network (CNN) for quantitative analysis. Methods: Retinal OCT-A with a 6x6mm field of view (FOV) were acquired using a Zeiss PlexElite. Multiple-volume acquisition and averaging enhanced the vessel network contrast used for training…
University of Warwick（华威大学）；Griffith University（格里菲斯大学）；University of Adelaide（阿德莱德大学）；King’s College London（伦敦国王学院）
The precise detection of blood vessels in retinal images is crucial to the early diagnosis of the retinal vascular diseases, e.g., diabetic, hypertensive and solar retinopathies. Existing works often fail in predicting the abnormal areas, e.g, sudden brighter and darker areas and are inclined to predict a pixel to background due to the significant class imbalance, leading to high accuracy and specificity while low sensitivity. To that end, we propose a novel error attention refining network (ERA-Net) that is capable of learning and predicting the potential false predictions in a two-stage mann…
Peking University（北京大学）；Peking University First Hospital（北京大学第一医院）；MGH/BWH Center for Clinical Data Science（麻省总医院/布莱根妇女医院临床数据科学中心）
Segmenting coronary arteries is challenging, as classic unsupervised methods fail to produce satisfactory results and modern supervised learning (deep learning) requires manual annotation which is often time-consuming and can some time be infeasible. To solve this problem, we propose a knowledge transfer based shape-consistent generative adversarial network (SC-GAN), which is an annotation-free approach that uses the knowledge from publicly available annotated fundus dataset to segment coronary arteries. The proposed network is trained in an end-to-end fashion, generating and segmenting synthe…
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Jiangxi Normal University（江西师范大学）
The precise segmentation of retinal blood vessels is of great significance for early diagnosis of eye-related diseases such as diabetes and hypertension. In this work, we propose a lightweight network named Spatial Attention U-Net (SA-UNet) that does not require thousands of annotated training samples and can be utilized in a data augmentation manner to use the available annotated samples more efficiently. SA-UNet introduces a spatial attention module which infers the attention map along the spatial dimension, and multiplies the attention map by the input feature map for adaptive feature refin…
University of Nevada, Reno（内华达大学里诺分校）；Deakin University（迪肯大学）；Houston Eye Associates（休斯顿眼科协会）；University of Nevada School of Medicine（内华达大学医学院）
Carrying out clinical diagnosis of retinal vascular degeneration using Fluorescein Angiography (FA) is a time consuming process and can pose significant adverse effects on the patient. Angiography requires insertion of a dye that may cause severe adverse effects and can even be fatal. Currently, there are no non-invasive systems capable of generating Fluorescein Angiography images. However, retinal fundus photography is a non-invasive imaging technique that can be completed in a few seconds. In order to eliminate the need for FA, we propose a conditional generative adversarial network (GAN) to…
Multimodal Deep Learning for Stroke Prediction and Detection using Retinal Imaging and Clinical Data
Division of Engineering, New York University Abu Dhabi（纽约大学阿布扎赫尔分校工程系）；Institute for Healthier Living Abu Dhabi（阿布扎赫尔健康生活研究所）；Eye Institute at Cleveland Clinic Abu Dhabi（阿布扎赫尔克利夫兰医学中心眼科研究所）；Canberra Hospital（堪培拉医院）
A General Bézier Tree Encoding Counterfactual Framework for Retinal-Vessel-Mediated Disease Analysis
Department of Electronic and Electrical Engineering（电子与电气工程系）；School of Data Science（数据科学学院）；Research Institute of Electrical Communication（电气通信研究所）；J. Crayton Pruitt Family Department of Biomedical Engineering（姜·克雷顿·普瑞特家庭生物医学工程系）
Fujian Agriculture and Forestry University（福建农林大学）；Digital Fujian Research Institute of Big Data for Agriculture and Forestry（数字福建农林大数据研究院）
Accurate segmentation of the blood vessels of the retina is an important step in clinical diagnosis of ophthalmic diseases. Many deep learning frameworks have come up for retinal blood vessels segmentation tasks. However, the complex vascular structure and uncertain pathological features make the blood vessel segmentation still very challenging. A novel U-shaped network named Multi-module Concatenation which is based on Atrous convolution and multi-kernel pooling is put forward to retinal vessels segmentation in this paper. The proposed network structure retains three layers the essential stru…
Generative artificial intelligence in ophthalmology: multimodal retinal images for the diagnosis of Alzheimer's disease with convolutional neural networks
Shadow artifacts from retinal vessels hinder the development of quantitative choroidal biomarkers in Optical Coherence Tomography (OCT), which limits the clinical applications of the choroidal OCT to the measures of layer thickness. In this paper, we present a novel framework that could eliminate the retinal shadows and realize the quantification of choroidal vasculature. Different from existing methods, we convert the shadow elimination into an object removal task, which can be handled by image inpainting techniques. We adopt and finetune a two-stage generative inpainting network which connec…
University of Calgary（卡尔加里大学）；Istanbul Medipol University（伊斯坦布尔梅迪波尔大学）；University of Southern Denmark（南丹麦大学）
The vascular structure of blood vessels is important in diagnosing retinal conditions such as glaucoma and diabetic retinopathy. Accurate segmentation of these vessels can help in detecting retinal objects such as the optic disc and optic cup and hence determine if there are damages to these areas. Moreover, the structure of the vessels can help in diagnosing glaucoma. The rapid development of digital imaging and computer-vision techniques has increased the potential for developing approaches for segmenting retinal vessels. In this paper, we propose an approach for segmenting retinal vessels t…
Optical coherence tomography (OCT) is a prevalent imaging technique for retina. However, it is affected by multiplicative speckle noise that can degrade the visibility of essential anatomical structures, including blood vessels and tissue layers. Although averaging repeated B-scan frames can significantly improve the signal-to-noise-ratio (SNR), this requires longer acquisition time, which can introduce motion artifacts and cause discomfort to patients. In this study, we propose a learning-based method that exploits information from the single-frame noisy B-scan and a pseudo-modality that is c…
FS-Net: Full Scale Network and Adaptive Threshold for Improving Extraction of Micro-Retinal Vessel Structures
Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术研究院（斯科尔泰克））；Artificial Intelligence Research Institute (AIRI)（人工智能研究院（AIRI））；University of Sharjah（沙迦大学）；Bian Que AI（扁鹊人工智能）；University College London（伦敦大学学院）；Research Institute for Medical and Health Sciences（医学与健康科学研究院）
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
Unsupervised domain adaptation has been proposed recently to tackle the so-called domain shift between training data and test data with different distributions. However, most of them only focus on single-target domain adaptation and cannot be applied to the scenario with multiple target domains. In this paper, we propose RVms, a novel unsupervised multi-target domain adaptation approach to segment retinal vessels (RVs) from multimodal and multicenter retinal images. RVms mainly consists of a style augmentation and transfer (SAT) module and a dual-teacher knowledge distillation (DTKD) module. S…
Southern University of Science and Technology（南方科技大学）；Shenzhen People’s Hospital（深圳市人民医院）；Jinan University（暨南大学）
Vascular segmentation extracts blood vessels from images and serves as the basis for diagnosing various diseases, like ophthalmic diseases. Ophthalmologists often require high-resolution segmentation results for analysis, which leads to super-computational load by most existing methods. If based on low-resolution input, they easily ignore tiny vessels or cause discontinuity of segmented vessels. To solve these problems, the paper proposes an algorithm named SuperVessel, which gives out high-resolution and accurate vessel segmentation using low-resolution images as input. We first take super-re…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Stein Eye Institute（斯坦眼科研究所）；VoxelCloud, Inc.（体素云公司）
The retinal vasculature provides important clues in the diagnosis and monitoring of systemic diseases including hypertension and diabetes. The microvascular system is of primary involvement in such conditions, and the retina is the only anatomical site where the microvasculature can be directly observed. The objective assessment of retinal vessels has long been considered a surrogate biomarker for systemic vascular diseases, and with recent advancements in retinal imaging and computer vision technologies, this topic has become the subject of renewed attention. In this paper, we present a novel…
SLOctolyzer: Fully automatic analysis toolkit for segmentation and feature extracting in scanning laser ophthalmoscopy images
University of Edinburgh（爱丁堡大学）；Institute for Regeneration and Repair, University of Edinburgh（爱丁堡大学再生与修复研究所）；Centre for Medical Informatics, University of Edinburgh（爱丁堡大学医学信息学中心）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Clinical Research Facility and Imaging, University of Edinburgh（爱丁堡大学临床研究设施与影像中心）；School of Mathematics, University of Edinburgh（爱丁堡大学数学学院）；Institute for Adaptive and Neural Computation, School of Informatics, University of Edinburgh（爱丁堡大学信息学院自适应与神经计算研究所）；Centre for Clinical Brain Sciences, University of Edinburgh（爱丁堡大学临床脑科学中心）
OCTCube-M: A 3D multimodal optical coherence tomography foundation model for retinal and systemic diseases with cross-cohort and cross-device validation
