Contrastive learning, a prominent approach within self-supervised learning, has demonstrated significant effectiveness in developing generalizable models for various applications involving natural images. However, recent research indicates that these successes do not necessarily extend to the medical imaging domain. In this paper, we investigate the reasons for this suboptimal performance and hypothesize that the dense distribution of medical images poses challenges to the pretext tasks in contrastive learning, particularly in constructing positive and negative pairs. We explore model performance under different augmentation strategies and compare the results to those achieved with strong augmentations. Our study includes six publicly available datasets covering multiple clinically relevant tasks. We further assess the model's generalizability through external evaluations. The model pre-trained with weak augmentation outperforms those with strong augmentation, improving AUROC from 0.838 to 0.848 and AUPR from 0.523 to 0.597 on MESSIDOR2, and showing similar enhancements across other datasets. Our findings suggest that optimizing the scale of augmentation is critical for enhancing the efficacy of contrastive learning in medical imaging.
Metadata-enhanced contrastive learning from retinal optical coherence tomography images
Imperial College London（帝国理工学院）；Medical University of Vienna（维也纳医科大学）；University of Michigan（密歇根大学）；King’s College London（伦敦国王学院）；Institute of Molecular and Clinical Ophthalmology Basel（巴塞尔分子与临床眼科研究所）；Universitat Basel（巴塞尔大学）；University College London（伦敦大学学院）；Moorfields Eye Hospital（摩尔菲尔兹眼科医院）；University of Southampton（南安普顿大学）
We explore the value of weak labels in learning transferable representations for medical images. Compared to hand-labeled datasets, weak or inexact labels can be acquired in large quantities at significantly lower cost and can provide useful training signals for data-hungry models such as deep neural networks. We consider weak labels in the form of pseudo-labels and propose a semi-weakly supervised contrastive learning (SWCL) framework for representation learning using semi-weakly annotated images. Specifically, we train a semi-supervised model to propagate labels from a small dataset consisti…
Recent contrastive learning methods achieved state-of-the-art in low label regimes. However, the training requires large batch sizes and heavy augmentations to create multiple views of an image. With non-contrastive methods, the negatives are implicitly incorporated in the loss, allowing different images and modalities as pairs. Although the meta-information (i.e., age, sex) in medical imaging is abundant, the annotations are noisy and prone to class imbalance. In this work, we exploited already existing temporal information (different visits from a patient) in a longitudinal optical coherence…
REVEAL++: Differentiable Phenotypic Grouping for Vision-Language Retinal Modeling of Alzheimer's Disease Risk
University of Virginia（弗吉尼亚大学）；J. Crayton Pruitt Family Department of Biomedical Engineering, Herbert Wertheim College of Engineering, University of Florida（佛罗里达大学赫伯特·韦特海姆工程学院J. Crayton Pruitt家庭生物医学工程系）
Monash University（莫纳什大学）；Griffith University（格里菲斯大学）；Beihang University（北京航空航天大学）；Zhejiang Normal University（浙江师范大学）；Alibaba Group（阿里巴巴集团）
Graph representation learning (GRL) is critical for graph-structured data analysis. However, most of the existing graph neural networks (GNNs) heavily rely on labeling information, which is normally expensive to obtain in the real world. Although some existing works aim to effectively learn graph representations in an unsupervised manner, they suffer from certain limitations, such as the heavy reliance on monotone contrastiveness and limited scalability. To overcome the aforementioned problems, we introduce a novel self-supervised graph representation learning algorithm via Graph Contrastive A…
Schepens Eye Research Institute of Massachusetts Eye and Ear（麻省眼耳医院谢本斯眼科研究所）；Harvard Medical School（哈佛医学院）；Eye and Vision Research Institute, Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院眼与视觉研究所）；Massachusetts Eye and Ear（麻省眼耳医院）
Ophthalmic images and derivatives such as the retinal nerve fiber layer (RNFL) thickness map are crucial for detecting and monitoring ophthalmic diseases (e.g., glaucoma). For computer-aided diagnosis of eye diseases, the key technique is to automatically extract meaningful features from ophthalmic images that can reveal the biomarkers (e.g., RNFL thinning patterns) linked to functional vision loss. However, representation learning from ophthalmic images that links structural retinal damage with human vision loss is non-trivial mostly due to large anatomical variations between patients. The ta…
REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Prediction
J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida（佛罗里达大学J. Crayton Pruitt家族生物医学工程系）；Research Institute of Electrical Communication, Tohoku University（东北大学电气通信研究所）；Department of Applied Mathematics & Statistics, Stony Brook University（石溪大学应用数学与统计学系）；Department of Computer Science, Stony Brook University（石溪大学计算机科学系）
PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysis
Christian Doppler Laboratory for Artificial Intelligence in Retina（人工智能在视网膜领域的克里斯蒂安·多普勒实验室）；Institute of Artificial Intelligence（人工智能研究所）；Center for Medical Data Science（医学数据科学中心）；Medical University of Vienna（维也纳医学大学）
OCTCube-M: A 3D multimodal optical coherence tomography foundation model for retinal and systemic diseases with cross-cohort and cross-device validation
3DTINC: Time-Equivariant Non-Contrastive Learning for Predicting Disease Progression from Longitudinal OCTs
Medical University of Vienna（维也纳医科大学）；BioMedIA, Imperial College London（伦敦帝国理工学院生物医学图像分析研究所）；Institute for AI and Informatics in Medicine, Klinikum rechts der Isar, Technical University Munich（慕尼黑工业大学伊萨尔河畔医院医学人工智能与信息学研究所）；Department of Ophthalmology, University of Basel（巴塞尔大学眼科系）
MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院眼科部）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research（科技研究局高性能计算研究所）
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Glaucoma is one of the ophthalmic diseases that may cause blindness, for which early detection and treatment are very important. Fundus images and optical coherence tomography (OCT) images are both widely-used modalities in diagnosing glaucoma. However, existing glaucoma grading approaches mainly utilize a single modality, ignoring the complementary information between fundus and OCT. In this paper, we propose an efficient multi-modality supervised contrastive learning framework, named COROLLA, for glaucoma grading. Through layer segmentation as well as thickness calculation and projection, re…
QMix: Quality-aware Learning with Mixed Noise for Robust Retinal Disease Diagnosis
Hong Kong University of Science and Technology（香港科技大学）；Fudan University（复旦大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Pretrained Deep 2.5D Models for Efficient Predictive Modeling from Retinal OCT
University of Basel（巴塞尔大学）；Technical University Munich（慕尼黑工业大学）；Medical University of Vienna（维也纳医科大学）；Moorfields Eye Hospital NHS Foundation Trust（摩尔菲尔斯眼科医院NHS基金会信托）；Imperial College London（帝国理工学院）；Institute of Molecular and Clinical Ophthalmology Basel（巴塞尔分子与临床眼科研究所）；University of Southampton（南安普顿大学）
ConKeD++ -- Improving descriptor learning for retinal image registration: A comprehensive study of contrastive losses
Instituto de Investigacion Biomédica de A Coruña (INIBIC)（拉科鲁尼亚生物医学研究所）；Universidade da Coruña（拉科鲁尼亚大学）；Departamento de Ciencias de la Computación y Tecnologías de la Información（计算机科学与信息技术系）；Grupo VARPA（VARPA团队）
Propagating Structural Guidance: Synthesizing Fluorescein Angiography from Fundus Images and Sparse OCT Scans
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education（教育部新一代人工智能技术及其跨学科应用重点实验室）；Tianyuan Honors School, Nanjing Medical University（南京医科大学天元荣誉学院）；Nanjing University of Science and Technology（南京理工大学）；Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院眼科）
University of Electronic Science and Technology of China（电子科技大学）；Shanghai AI Lab（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；ShanghaiTech University（上海科技大学）；West China Hospital, Sichuan University（四川大学华西医院）
Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance for medical image segmentation, yet need plenty of manual annotations for training. Semi-Supervised Learning (SSL) methods are promising to reduce the requirement of annotations, but their performance is still limited when the dataset size and the number of annotated images are small. Leveraging existing annotated datasets with similar anatomical structures to assist training has a potential for improving the model's performance. However, it is further challenged by the cross-anatomy domain shift due to the differe…
Deep-learning-based clustering of OCT images for biomarker discovery in age-related macular degeneration (Pinnacle study report 4)
BioMedIA, Imperial College London（伦敦帝国理工学院BioMedIA）；University of Southampton（南安普顿大学）；University College London（伦敦大学学院）；Moorfields Eye Hospital（摩尔菲尔德眼科医院）；Medical University of Vienna（维也纳医科大学）；Christian Doppler Forschungsgesellschaft（克里斯蒂安多普勒研究协会）；University of Basel（巴塞尔大学）；Technical University Munich（慕尼黑工业大学）
eMargin: Revisiting Contrastive Learning with Margin-Based Separation
Department of Computer Science, Norwegian University of Science and Technology, Norway（挪威科学技术大学计算机科学系）；Department of Computer Science, United States Naval Academy, USA（美国海军学院计算机科学系）
Understanding Contrastive Learning via Distributionally Robust Optimization
University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）；Institute of Dataspace, Hefei Comprehensive National Science Center（合肥综合性国家科学中心数据空间研究院）
VARPA Research Group, Instituto de Investigación Biomédica de A Coruña (INIBIC), Universidade da Coruña（拉科鲁尼亚大学生物医学研究所（INIBIC）VARPA研究组）；Centro de Investigación CITIC, Universidade da Coruña（拉科鲁尼亚大学CITIC研究中心）
In ophthalmology, the study of the retinal microcirculation is a key issue in the analysis of many ocular and systemic diseases, like hypertension or diabetes. This motivates the research on improving the retinal vasculature segmentation. Nowadays, Fully Convolutional Neural Networks (FCNs) usually represent the most successful approach to image segmentation. However, the success of these models is conditioned by an adequate selection and adaptation of the architectures and techniques used, as well as the availability of a large amount of annotated data. These two issues become specially relev…
Hangzhou Dianzi University（杭州电子科技大学）；the Second Affiliated Hospital of Zhejiang University（浙江大学医学院附属第二医院）；Waseda University（早稻田大学）；Communication University of Zhejiang（浙江传媒学院）
Precise and rapid categorization of images in the B-scan ultrasound modality is vital for diagnosing ocular diseases. Nevertheless, distinguishing various diseases in ultrasound still challenges experienced ophthalmologists. Thus a novel contrastive disentangled network (CDNet) is developed in this work, aiming to tackle the fine-grained image categorization (FGIC) challenges of ocular abnormalities in ultrasound images, including intraocular tumor (IOT), retinal detachment (RD), posterior scleral staphyloma (PSS), and vitreous hemorrhage (VH). Three essential components of CDNet are the weakl…
Helmholtz Zentrum München（慕尼黑亥姆霍兹中心）；Technical University of Munich（慕尼黑工业大学）
Fundus photography is the primary method for retinal imaging and essential for diabetic retinopathy prevention. Automated segmentation of fundus photographs would improve the quality, capacity, and cost-effectiveness of eye care screening programs. However, current segmentation methods are not robust towards the diversity in imaging conditions and pathologies typical for real-world clinical applications. To overcome these limitations, we utilized contrastive self-supervised learning to exploit the large variety of unlabeled fundus images in the publicly available EyePACS dataset. We pre-traine…
Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong（香港中文大学医学智能与扩展现实研究所）
F Hoffmann-La Roche AG（罗氏公司）；Genentech Inc（基因泰克公司）；ETH Zurich（苏黎世联邦理工学院）；Uppsala University（乌普萨拉大学）
Accurate segmentation of retinal fluids in 3D Optical Coherence Tomography images is key for diagnosis and personalized treatment of eye diseases. While deep learning has been successful at this task, trained supervised models often fail for images that do not resemble labeled examples, e.g. for images acquired using different devices. We hereby propose a novel semi-supervised learning framework for segmentation of volumetric images from new unlabeled domains. We jointly use supervised and contrastive learning, also introducing a contrastive pairing scheme that leverages similarity between nea…
UnShadowNet: Illumination Critic Guided Contrastive Learning For Shadow Removal
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；University of Limerick（利默里克大学）；Lero (the Science Foundation Ireland Research Centre for Software)（Lero（爱尔兰科学基金会软件研究中心））；Indian Statistical Institute（印度统计研究所）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shandong University of Science and Technology（山东科技大学）
Efficient medical image segmentation aims to provide accurate pixel-wise predictions for medical images with a lightweight implementation framework. However, lightweight frameworks generally fail to achieve superior performance and suffer from poor generalizable ability on cross-domain tasks. In this paper, we explore the generalizable knowledge distillation for the efficient segmentation of cross-domain medical images. Considering the domain gaps between different medical datasets, we propose the Model-Specific Alignment Networks (MSAN) to obtain the domain-invariant representations. Meanwhil…
Neural networks pre-trained on a self-supervision scheme have become the standard when operating in data rich environments with scarce annotations. As such, fine-tuning a model to a downstream task in a parameter-efficient but effective way, e.g. for a new set of classes in the case of semantic segmentation, is of increasing importance. In this work, we propose and investigate several contributions to achieve a parameter-efficient but effective adaptation for semantic segmentation on two medical imaging datasets. Relying on the recently popularized prompt tuning approach, we provide a prompt-a…
