Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging
Cluster of Excellence “Machine Learning”（卓越集群“机器学习”）；University of Tübingen（蒂宾根大学）；Faculty of Health Sciences and Medicine, University of Lucerne（卢塞恩大学健康科学与医学院）
Stefano Woerner, Christian F. Baumgartner
备注：
Accepted as an oral presentation in MICCAI 2024 2nd International Workshop on Foundation Models for General Medical AI
Data scarcity is a major limiting factor for applying modern machine learning techniques to clinical tasks. Although sufficient data exists for some well-studied medical tasks, there remains a long tail of clinically relevant tasks with poor data availability. Recently, numerous foundation models have demonstrated high suitability for few-shot learning (FSL) and zero-shot learning (ZSL), potentially making them more accessible to practitioners. However, it remains unclear which foundation model performs best on FSL medical image analysis tasks and what the optimal methods are for learning from limited data. We conducted a comprehensive benchmark study of ZSL and FSL using 16 pretrained foundation models on 19 diverse medical imaging datasets. Our results indicate that BiomedCLIP, a model pretrained exclusively on medical data, performs best on average for very small training set sizes, while very large CLIP models pretrained on LAION-2B perform best with slightly more training samples. However, simply fine-tuning a ResNet-18 pretrained on ImageNet performs similarly with more than five training examples per class. Our findings also highlight the need for further research on foundation models specifically tailored for medical applications and the collection of more datasets to train these models.
Due to the high human cost of annotation, it is non-trivial to curate a large-scale medical dataset that is fully labeled for all classes of interest. Instead, it would be convenient to collect multiple small partially labeled datasets from different matching sources, where the medical images may have only been annotated for a subset of classes of interest. This paper offers an empirical understanding of an under-explored problem, namely partially supervised multi-label classification (PSMLC), where a multi-label classifier is trained with only partially labeled medical images. In contrast to …
Navigating Distribution Shifts in Medical Image Analysis: A Survey
Life Simulation Research Center, Beijing Academy of Artificial Intelligence（北京人工智能生命模拟研究中心）；Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology（王国阿卜杜勒·阿齐兹国王科技大学电气与数学科学与工程系）；Department of Intelligent Science, School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学先进科技学院智能科学系）；Computer Science, School of Computer Science and Informatics, University of Liverpool（利物浦大学计算机科学与信息学学院）；SDAIA-KFUPM Joint Research Centre for Artificial Intelligence, King Fahd University of Petroleum and Minerals（法赫德石油与矿物大学人工智能SDAIA-KFUPM联合研究中心）；Nuffield Department of Primary Care Health Sciences, University of Oxford（牛津大学初级保健健康科学努尔菲尔德部门）
Jordan University of Science and Technology（约旦科技大学）
The scarcity of high-quality annotated medical imaging datasets is a major problem that collides with machine learning applications in the field of medical imaging analysis and impedes its advancement. Self-supervised learning is a recent training paradigm that enables learning robust representations without the need for human annotation which can be considered an effective solution for the scarcity of annotated medical data. This article reviews the state-of-the-art research directions in self-supervised learning approaches for image data with a concentration on their applications in the fiel…
A Systematic Review of Few-Shot Learning in Medical Imaging
Institute of Information Science and Technologies “Alessandro Faedo”, National Research Council of Italy (ISTI-CNR)（意大利国家研究委员会信息科学与技术研究所）；Department of Information Engineering, University of Pisa（比萨大学信息工程系）
Medical image classification is often challenging for two reasons: a lack of labelled examples due to expensive and time-consuming annotation protocols, and imbalanced class labels due to the relative scarcity of disease-positive individuals in the wider population. Semi-supervised learning (SSL) methods exist for dealing with a lack of labels, but they generally do not address the problem of class imbalance. In this study we propose Adaptive Blended Consistency Loss (ABCL), a drop-in replacement for consistency loss in perturbation-based SSL methods. ABCL counteracts data skew by adaptively m…
Imperial College London（帝国理工学院）；Alan Turing Institute（艾伦·图灵研究所）；TECNALIA（泰克纳利亚研究中心）；Basque Research & Technology Alliance (BRTA)（巴斯克研究与技术联盟（BRTA））
Data quality is the key factor for the development of trustworthy AI in healthcare. A large volume of curated datasets with controlled confounding factors can help improve the accuracy, robustness and privacy of downstream AI algorithms. However, access to good quality datasets is limited by the technical difficulty of data acquisition and large-scale sharing of healthcare data is hindered by strict ethical restrictions. Data synthesis algorithms, which generate data with a similar distribution as real clinical data, can serve as a potential solution to address the scarcity of good quality dat…
Supervised learning has proved effective for medical image analysis. However, it can utilize only the small labeled portion of data; it fails to leverage the large amounts of unlabeled data that is often available in medical image datasets. Supervised models are further handicapped by domain shifts, when the labeled dataset, despite being large enough, fails to cover different protocols or ethnicities. In this paper, we introduce \emph{extreme consistency}, which overcomes the above limitations, by maximally leveraging unlabeled data from the same or a different domain in a teacher-student sem…
The scarcity of pixel-level annotation is a prevalent problem in medical image segmentation tasks. In this paper, we introduce a novel regularization strategy involving interpolation-based mixing for semi-supervised medical image segmentation. The proposed method is a new consistency regularization strategy that encourages segmentation of interpolation of two unlabelled data to be consistent with the interpolation of segmentation maps of those data. This method represents a specific type of data-adaptive regularization paradigm which aids to minimize the overfitting of labelled data under high…
TU Darmstadt（达姆施塔特工业大学）；TU Munich（慕尼黑工业大学）；Radboud University Medical Center（拉德堡德大学医学中心）
Generative Adversarial Networks (GANs) and their extensions have carved open many exciting ways to tackle well known and challenging medical image analysis problems such as medical image de-noising, reconstruction, segmentation, data simulation, detection or classification. Furthermore, their ability to synthesize images at unprecedented levels of realism also gives hope that the chronic scarcity of labeled data in the medical field can be resolved with the help of these generative models. In this review paper, a broad overview of recent literature on GANs for medical applications is given, th…
Medical imaging AI systems such as disease classification and segmentation are increasingly inspired and transformed from computer vision based AI systems. Although an array of adversarial training and/or loss function based defense techniques have been developed and proved to be effective in computer vision, defending against adversarial attacks on medical images remains largely an uncharted territory due to the following unique challenges: 1) label scarcity in medical images significantly limits adversarial generalizability of the AI system; 2) vastly similar and dominant fore- and backgroun…
This article discusses how the language of causality can shed new light on the major challenges in machine learning for medical imaging: 1) data scarcity, which is the limited availability of high-quality annotations, and 2) data mismatch, whereby a trained algorithm may fail to generalize in clinical practice. Looking at these challenges through the lens of causality allows decisions about data collection, annotation procedures, and learning strategies to be made (and scrutinized) more transparently. We discuss how causal relationships between images and annotations can not only have profound…
Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Imperial College London（帝国理工学院）
Convolutional neural networks have been applied to a wide variety of computer vision tasks. Recent advances in semantic segmentation have enabled their application to medical image segmentation. While most CNNs use two-dimensional kernels, recent CNN-based publications on medical image segmentation featured three-dimensional kernels, allowing full access to the three-dimensional structure of medical images. Though closely related to semantic segmentation, medical image segmentation includes specific challenges that need to be addressed, such as the scarcity of labelled data, the high class imb…
University of Oxford（牛津大学）；UiT The Arctic University of Norway（挪威北极大学）
Using decentralized data for federated training is one promising emerging research direction for alleviating data scarcity in the medical domain. However, in contrast to large-scale fully labeled data commonly seen in general object recognition tasks, the local medical datasets are more likely to only have images annotated for a subset of classes of interest due to high annotation costs. In this paper, we consider a practical yet under-explored problem, where underrepresented classes only have few labeled instances available and only exist in a few clients of the federated system. We show that…
This review presents a comprehensive study on the role of GANs in addressing the challenges related to COVID-19 data scarcity and diagnosis. It is the first review that summarizes the different GANs methods and the lungs images datasets for COVID-19. It attempts to answer the questions related to applications of GANs, popular GAN architectures, frequently used image modalities, and the availability of source code. This review included 57 full-text studies that reported the use of GANs for different applications in COVID-19 lungs images data. Most of the studies (n=42) used GANs for data augmen…
Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma（人工智能与计算机系统单位，工程系，罗马生物医学大学）；College of Computer Science and Software Engineering, Shenzhen University（计算机科学与软件工程学院，深圳大学）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University（诊断与介入系，辐射物理，生物医学工程，乌梅大学）
FUJIFILM Corporation（富士胶片公司）；AWS AI Labs（AWS人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）；Stanford University（斯坦福大学）
Learning visual representations of medical images (e.g., X-rays) is core to medical image understanding but its progress has been held back by the scarcity of human annotations. Existing work commonly relies on fine-tuning weights transferred from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. Meanwhile, several recent studies show exciting results from unsupervised contrastive learning from natural images, but we fin…
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）；School of Medicine, Emory University（埃默里大学医学院）
Motivation: Medical image analysis involves tasks to assist physicians in qualitative and quantitative analysis of lesions or anatomical structures, significantly improving the accuracy and reliability of diagnosis and prognosis. Traditionally, these tasks are finished by physicians or medical physicists and lead to two major problems: (i) low efficiency; (ii) biased by personal experience. In the past decade, many machine learning methods have been applied to accelerate and automate the image analysis process. Compared to the enormous deployments of supervised and unsupervised learning models…
Politecnico di Torino（都灵理工大学）；Istituto Italiano di Tecnologia（意大利理工学院）
Deep learning has thrived by training on large-scale datasets. However, in many applications, as for medical image diagnosis, getting massive amount of data is still prohibitive due to privacy, lack of acquisition homogeneity and annotation cost. In this scenario, transfer learning from natural image collections is a standard practice that attempts to tackle shape, texture and color discrepancies all at once through pretrained model fine-tuning. In this work, we propose to disentangle those challenges and design a dedicated network module that focuses on color adaptation. We combine learning f…
Medical image classification is one of the most critical problems in the image recognition area. One of the major challenges in this field is the scarcity of labelled training data. Additionally, there is often class imbalance in datasets as some cases are very rare to happen. As a result, accuracy in classification task is normally low. Deep Learning models, in particular, show promising results on image segmentation and classification problems, but they require very large datasets for training. Therefore, there is a need to generate more of synthetic samples from the same distribution. Previ…
Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Lehigh University（里海大学）
针对医学图像数据稀缺及现有方法难以准确呈现解剖与病理细节的问题，提出结合解剖-病理提示和视觉码本细粒度图文对齐的合成模型，并在胸部 X 射线数据上验证其语义准确性。
Text-guided Foundation Model Adaptation for Long-Tailed Medical Image Classification
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；University of British Columbia（不列颠哥伦比亚大学）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
Semise: Semi-supervised learning for severity representation in medical image
Hanoi University of Science and Technology（河内理工大学）；University of Southern California（南加州大学）；VinUni-Illinois Smart Health Center, VinUniversity（VinUniversity-VinUniversity伊利诺伊智能健康中心）
Stony Brook University（石溪大学）；Jadavpur University（贾达普大学）；University of Toronto（多伦多大学）；Kyungpook National University（庆北国立大学）
Due to the scarcity of labeled data, Contrastive Self-Supervised Learning (SSL) frameworks have lately shown great potential in several medical image analysis tasks. However, the existing contrastive mechanisms are sub-optimal for dense pixel-level segmentation tasks due to their inability to mine local features. To this end, we extend the concept of metric learning to the segmentation task, using a dense (dis)similarity learning for pre-training a deep encoder network, and employing a semi-supervised paradigm to fine-tune for the downstream task. Specifically, we propose a simple convolutiona…
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
The goal of this paper is to interactively refine the automatic segmentation on challenging structures that fall behind human performance, either due to the scarcity of available annotations or the difficulty nature of the problem itself, for example, on segmenting cancer or small organs. Specifically, we propose a novel Transformer-based architecture for Interactive Segmentation (TIS), that treats the refinement task as a procedure for grouping pixels with similar features to those clicks given by the end users. Our proposed architecture is composed of Transformer Decoder variants, which natu…
