OCTCube-M: A 3D multimodal optical coherence tomography foundation model for retinal and systemic diseases with cross-cohort and cross-device validation
Zixuan Liu, Hanwen Xu, Addie Woicik, Linda G. Shapiro, Marian Blazes, Yue Wu, Verena Steffen, Catherine Cukras, Cecilia S. Lee, Miao Zhang, Aaron Y. Lee, Sheng Wang
We present OCTCube-M, a 3D OCT-based multi-modal foundation model for jointly analyzing OCT and en face images. OCTCube-M first developed OCTCube, a 3D foundation model pre-trained on 26,685 3D OCT volumes encompassing 1.62 million 2D OCT images. It then exploits a novel multi-modal contrastive learning framework COEP to integrate other retinal imaging modalities, such as fundus autofluorescence and infrared retinal imaging, into OCTCube, efficiently extending it into multi-modal foundation models. OCTCube achieves best performance on predicting 8 retinal diseases, demonstrating strong generalizability on cross-cohort, cross-device and cross-modality prediction. OCTCube can also predict cross-organ nodule malignancy (CT) and low cardiac ejection fraction as well as systemic diseases, such as diabetes and hypertension, revealing its wide applicability beyond retinal diseases. We further develop OCTCube-IR using COEP with 26,685 OCT and IR image pairs. OCTCube-IR can accurately retrieve between OCT and IR images, allowing joint analysis between 3D and 2D retinal imaging modalities. Finally, we trained a tri-modal foundation model OCTCube-EF from 4 million 2D OCT images and 400K en face retinal images. OCTCube-EF attains the best performance on predicting the growth rate of geographic atrophy (GA) across datasets collected from 6 multi-center global trials conducted in 23 countries. This improvement is statistically equivalent to running a clinical trial with more than double the size of the original study. Our analysis based on another retrospective case study reveals OCTCube-EF's ability to avoid false positive Phase-III results according to its accurate treatment effect estimation on the Phase-II results. In sum, OCTCube-M is a 3D multi-modal foundation model framework that integrates OCT and other retinal imaging modalities revealing substantial diagnostic and prognostic benefits.
Explainable Artificial Intelligence (AI) in the form of an interpretable and semiautomatic approach to stage grading ocular pathologies such as Diabetic retinopathy, Hypertensive retinopathy, and other retinopathies on the backdrop of major systemic diseases. The experimental study aims to evaluate an explainable staged grading process without using deep Convolutional Neural Networks (CNNs) directly. Many current CNN-based deep neural networks used for diagnosing retinal disorders might have appreciable performance but fail to pinpoint the basis driving their decisions. To improve these decisi…
The Role of AI in Early Detection of Life-Threatening Diseases: A Retinal Imaging Perspective
Naif Arab University for Security Sciences（奈伊夫阿拉伯安全科学大学）；Charles Sturt University（查尔斯特大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Prediction
J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida（佛罗里达大学J. Crayton Pruitt家族生物医学工程系）；Research Institute of Electrical Communication, Tohoku University（东北大学电气通信研究所）；Department of Applied Mathematics & Statistics, Stony Brook University（石溪大学应用数学与统计学系）；Department of Computer Science, Stony Brook University（石溪大学计算机科学系）
A Disease-Specific Foundation Model Using Over 100K Fundus Images: Release and Validation for Abnormality and Multi-Disease Classification on Downstream Tasks
Seoul National University（首尔大学）；Seoul National University Hospital（首尔大学附属医院）
Explainable Multi-Task Retinal Imaging Reveals Microvascular Signals for Systemic Risk Stratification in Type 2 Diabetes: A Pilot Study
Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology（深圳先进技术大学计算机科学与人工智能学院）；Frontier Science Computing Center, Zhuhai Institute of Advanced Technology Chinese Academy of Sciences（中国科学院珠海先进技术研究院前沿科学计算中心）；Chinese University of Hong Kong（香港中文大学）；Zhuhai People's Hospital (The Affiliated Hospital of Beijing Institute of Technology, Zhuhai Clinical Medical College of Jinan University)（珠海人民医院（北京理工大学珠海临床医学院附属医院））；Lampang Inter-Tech College, Lampang Thailand（泰国 Lampang 职业技术学院）
JIS College of Engineering（JIS工程学院）；Kalyani Govt. Engineering College（卡利亚尼政府工程学院）
Segmentation of blood vessels in retinal images provides early diagnosis of diseases like glaucoma, diabetic retinopathy and macular degeneration. Among these diseases occurrence of Glaucoma is most frequent and has serious ocular consequences that can even lead to blindness, if it is not detected early. The clinical criteria for the diagnosis of glaucoma include intraocular pressure measurement, optic nerve head evaluation, retinal nerve fiber layer and visual field defects. This form of blood vessel segmentation helps in early detection for ophthalmic diseases, and potentially reduces the ri…
The status of retinal arteriovenous crossing is of great significance for clinical evaluation of arteriolosclerosis and systemic hypertension. As an ophthalmology diagnostic criteria, Scheie's classification has been used to grade the severity of arteriolosclerosis. In this paper, we propose a deep learning approach to support the diagnosis process, which, to the best of our knowledge, is one of the earliest attempts in medical imaging. The proposed pipeline is three-fold. First, we adopt segmentation and classification models to automatically obtain vessels in a retinal image with the corresp…
Prediction of Alzheimer's Disease Risk Factors from Retinal Images via Deep Learning: Development and Validation of Biologically Relevant Morphological Associations in the UK Biobank
J. Crayton Pruitt Family Dept. of Biomedical Engineering, University of Florida（朱·克雷顿·普瑞特生物医学工程系，佛罗里达大学）；University of Florida Research Computing（佛罗里达大学研究计算中心）；Meta AI (FAIR)（Meta AI（FAIR））；School of Behavioral and Brain Sciences, University of Texas at Dallas（德克萨斯大学达拉斯分校行为与脑科学学院）；Dept. of Electrical and Computer Engineering, University of Florida（佛罗里达大学电气与计算机工程系）；Dept. of Computer and Information Science and Engineering, University of Florida（佛罗里达大学计算机与信息科学与工程系）；Center for Cognitive Aging and Memory, University of Florida（佛罗里达大学认知衰老与记忆中心）
Machine Learning Prediction of Cardiovascular Risk in Type 1 Diabetes Mellitus Using Radiomics Features from Multimodal Retinal Images
Universitat Politècnica de Catalunya (UPC)（加泰罗尼亚理工大学）；Institut Clínic d´Oftalmología (ICOF)（眼科临床研究所）；Hospital Clínic de Barcelona（巴塞罗那临床医院）；Intelligent Data Science and Artificial Intelligence (IDEAI-UPC) Research Center（智能数据科学与人工智能研究中心）；August Pi i Sunyer Biomedical Research Institute (IDIBAPS)（奥古斯特·皮伊·苏涅尔生物医学研究所）；School of Medicine, Universitat de Barcelona（巴塞罗那大学医学院）
Pulsatile changes in retinal vascular geometry over the cardiac cycle have clinical implications for the diagnosis of ocular and systemic vascular diseases, including ischemia, coronary heart diseases, and diabetes mellitus and its complications. Thus, analysis of the pulsatile changes over the cardiac cycle is a potential non-invasive assessment for the presence of ocular and systemic vascular diseases. The cardiac rhythm influences these pulsatile changes in the retina, is a result of the change in blood volumetric flow entering the ophthalmic-vascular system under a certain level of intraoc…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Stein Eye Institute（斯坦眼科研究所）；VoxelCloud, Inc.（体素云公司）
The retinal vasculature provides important clues in the diagnosis and monitoring of systemic diseases including hypertension and diabetes. The microvascular system is of primary involvement in such conditions, and the retina is the only anatomical site where the microvasculature can be directly observed. The objective assessment of retinal vessels has long been considered a surrogate biomarker for systemic vascular diseases, and with recent advancements in retinal imaging and computer vision technologies, this topic has become the subject of renewed attention. In this paper, we present a novel…
The accurate retinal vessel segmentation (RVS) is of great significance to assist doctors in the diagnosis of ophthalmology diseases and other systemic diseases. Manually designing a valid neural network architecture for retinal vessel segmentation requires high expertise and a large workload. In order to improve the performance of vessel segmentation and reduce the workload of manually designing neural network, we propose novel approach which applies neural architecture search (NAS) to optimize an encoder-decoder architecture for retinal vessel segmentation. A modified evolutionary algorithm …
Systemic diseases such as diabetes, hypertension, atherosclerosis are among the leading causes of annual human mortality rate. It is suggested that retinal and conjunctival vascular tortuosity is a potential biomarker for such systemic diseases. Most importantly, it is observed that the tortuosity depends on the thickness of these vessels. Therefore, selective calculation of tortuosity within specific vessel thicknesses is required depending on the disease being analysed. In this paper, we propose a thickness sensitive vessel extraction framework that is primarily applicable for studies relate…
FusionFM: Fusing Eye-specific Foundational Models for Optimized Ophthalmic Diagnosis
Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore（新加坡国立大学 Yong Loo Lin 医学院眼科系）；Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore, Singapore（新加坡国立大学 Yong Loo Lin 医学院创新与精准眼科健康中心）；Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore（新加坡眼科研究院）；Ophthalmology and Visual Science Academic Clinical Program, Duke-NUS Medical School, Singapore, Singapore（杜克-国立新加坡大学医学学校眼科与视觉科学学术临床项目）；Centre for Medical Image Computing, University College London, London, United Kingdom（伦敦大学学院医学图像计算中心）；Joint Shantou International Eye Center, Shantou University and the Chinese University of Hong Kong, Shantou, China（汕尾国际眼科中心，汕尾大学及香港中文大学）；Department of Ophthalmology, Stadtspital Zürich, Zurich, Switzerland（苏黎世州立医院眼科部）；Spross Research Institute, Zurich, Switzerland（苏黎世 Spross 研究院）；Institute of High Performance Computing, A*STAR, Singapore（新加坡 A*STAR 高性能计算研究院）；NIHR Biomedical Research Centre, Moorfields Eye Hospital NHS Foundation Trust, London, United Kingdom（英国国家健康服务基金会 Moorfields 眼科医院生物医学研究中心）；Institute of Ophthalmology, University College London, London, United Kingdom（伦敦大学学院眼科研究院）
Youtu Lab, Tencent（腾讯优图实验室）；School of Information Science and Engineering, Xiamen University（厦门大学信息科学与工程学院）
Retinal artery/vein (A/V) classification plays a critical role in the clinical biomarker study of how various systemic and cardiovascular diseases affect the retinal vessels. Conventional methods of automated A/V classification are generally complicated and heavily depend on the accurate vessel segmentation. In this paper, we propose a multi-task deep neural network with spatial activation mechanism that is able to segment full retinal vessel, artery and vein simultaneously, without the pre-requirement of vessel segmentation. The input module of the network integrates the domain knowledge of w…
University of Rostock（罗斯托克大学）；Rostock University Medical Center（罗斯托克大学医学中心）
Advances in optical coherence tomography (OCT) have enabled noninvasive imaging of substructures of the human retina with high spatial resolution. OCT examinations are now a standard procedure in clinics and an integral part of ophthalmic research. The interpretation of the OCT helps ophthalmologists understand the impact of various retinal and systemic diseases on the structure of the retina in a way not previously possible. In the early stages of retinal diseases, however, the identification and analysis of small and localized substructural changes in the retina remains a challenge. We prese…
Arizona State University（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Mayo Clinic（梅奥诊所）
Real-world non-mydriatic retinal fundus photography is prone to artifacts, imperfections and low-quality when certain ocular or systemic co-morbidities exist. Artifacts may result in inaccuracy or ambiguity in clinical diagnoses. In this paper, we proposed a simple but effective end-to-end framework for enhancing poor-quality retinal fundus images. Leveraging the optimal transport theory, we proposed an unpaired image-to-image translation scheme for transporting low-quality images to their high-quality counterparts. We theoretically proved that a Generative Adversarial Networks (GAN) model wit…
VARPA Research Group, Instituto de Investigación Biomédica de A Coruña (INIBIC), Universidade da Coruña（拉科鲁尼亚大学生物医学研究所（INIBIC）VARPA研究组）；Centro de Investigación CITIC, Universidade da Coruña（拉科鲁尼亚大学CITIC研究中心）
In ophthalmology, the study of the retinal microcirculation is a key issue in the analysis of many ocular and systemic diseases, like hypertension or diabetes. This motivates the research on improving the retinal vasculature segmentation. Nowadays, Fully Convolutional Neural Networks (FCNs) usually represent the most successful approach to image segmentation. However, the success of these models is conditioned by an adequate selection and adaptation of the architectures and techniques used, as well as the availability of a large amount of annotated data. These two issues become specially relev…
FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification
Center for Artificial Intelligence and Robotics, New York University Abu Dhabi（纽约大学阿布扎比分校人工智能与机器人中心）；New York University（纽约大学）；Harvard University（哈佛大学）
A General Bézier Tree Encoding Counterfactual Framework for Retinal-Vessel-Mediated Disease Analysis
Department of Electronic and Electrical Engineering（电子与电气工程系）；School of Data Science（数据科学学院）；Research Institute of Electrical Communication（电气通信研究所）；J. Crayton Pruitt Family Department of Biomedical Engineering（姜·克雷顿·普瑞特家庭生物医学工程系）
National Taiwan University（台湾大学）；Chang Gung Memorial Hospital（长庚纪念医院）；Chang Gung University（长庚大学）
Automated vascular segmentation on optical coherence tomography angiography (OCTA) is important for the quantitative analyses of retinal microvasculature in neuroretinal and systemic diseases. Despite recent improvements, artifacts continue to pose challenges in segmentation. Our study focused on removing the speckle noise artifact from OCTA images when performing segmentation. Speckle noise is common in OCTA and is particularly prominent over large non-perfusion areas. It may interfere with the proper assessment of retinal vasculature. In this study, we proposed a novel Supervision Vessel Seg…
Domain-specific augmentations with resolution agnostic self-attention mechanism improves choroid segmentation in optical coherence tomography images
School of Mathematics, University of Edinburgh（爱丁堡大学数学学院）；Institute for Regeneration and Repair, University of Edinburgh（爱丁堡大学再生与修复研究所）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Centre for Medical Informatics, University of Edinburgh（爱丁堡大学医学信息学中心）；Clinical Research Facility and Imaging, University of Edinburgh（爱丁堡大学临床研究与影像中心）；Department of Vision Sciences, Glasgow Caledonian University（格拉斯哥卡利多尼安大学视觉科学系）；British Heart Foundation Centre for Cardiovascular Science, University of Edinburgh（爱丁堡大学英国心脏基金会心血管科学中心）；Centre for Clinical Brain Sciences, University of Edinburgh（爱丁堡大学临床脑科学中心）
Is an Ultra Large Natural Image-Based Foundation Model Superior to a Retina-Specific Model for Detecting Ocular and Systemic Diseases?
National University of Singapore（新加坡国立大学）；Northeastern University（东北大学）；University College London（伦敦大学学院）；NIHR Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust（莫菲尔德眼科医院 NHS 基金会信托 NIHR 生物医学研究中心）；Singapore Eye Research Institute, Singapore National Eye Centre（新加坡眼科研究所，新加坡国家眼科中心）；Institute of High-Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；The Chinese University of Hong Kong（香港中文大学）；Stadtspital Zürich（苏黎世市立医院）；Spross Research Institute（斯普罗斯研究所）；Semmelweis University（塞梅尔维斯大学）；Shantou University（汕头大学）；Shantou University Medical College（汕头大学医学院）
Generalist versus Specialist Vision Foundation Models for Ocular Disease and Oculomics
Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Hawkes Institute, University College London（伦敦大学学院霍克斯研究所）；NIHR Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust（莫菲尔德眼科医院 NHS 基金会信托 NIHR 生物医学研究中心）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院创新与精准眼健康中心）；Singapore Eye Research Institute, Singapore National Eye Centre（新加坡国家眼科中心新加坡眼科研究所）；Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院教育部人工智能重点实验室）；Stadtspital Zürich（苏黎世市立医院）；Spross Research Institute（斯普罗斯研究所）；Tsinghua Medicine, Tsinghua University（清华大学清华医学）；Beijing Visual Science and Translational Eye Research Institute, Beijing Tsinghua Changgung Hospital（北京清华长庚医院北京视觉科学与转化眼科研究所）；University College London（伦敦大学学院）
SLOctolyzer: Fully automatic analysis toolkit for segmentation and feature extracting in scanning laser ophthalmoscopy images
University of Edinburgh（爱丁堡大学）；Institute for Regeneration and Repair, University of Edinburgh（爱丁堡大学再生与修复研究所）；Centre for Medical Informatics, University of Edinburgh（爱丁堡大学医学信息学中心）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Clinical Research Facility and Imaging, University of Edinburgh（爱丁堡大学临床研究设施与影像中心）；School of Mathematics, University of Edinburgh（爱丁堡大学数学学院）；Institute for Adaptive and Neural Computation, School of Informatics, University of Edinburgh（爱丁堡大学信息学院自适应与神经计算研究所）；Centre for Clinical Brain Sciences, University of Edinburgh（爱丁堡大学临床脑科学中心）
