Medical Vision-Language Pretraining (MedVLP) shows promise in learning generalizable and transferable visual representations from paired and unpaired medical images and reports. MedVLP can provide useful features to downstream tasks and facilitate adapting task-specific models to new setups using fewer examples. However, existing MedVLP methods often differ in terms of datasets, preprocessing, and finetuning implementations. This pose great challenges in evaluating how well a MedVLP method generalizes to various clinically-relevant tasks due to the lack of unified, standardized, and comprehensive benchmark. To fill this gap, we propose BenchX, a unified benchmark framework that enables head-to-head comparison and systematical analysis between MedVLP methods using public chest X-ray datasets. Specifically, BenchX is composed of three components: 1) Comprehensive datasets covering nine datasets and four medical tasks; 2) Benchmark suites to standardize data preprocessing, train-test splits, and parameter selection; 3) Unified finetuning protocols that accommodate heterogeneous MedVLP methods for consistent task adaptation in classification, segmentation, and report generation, respectively. Utilizing BenchX, we establish baselines for nine state-of-the-art MedVLP methods and found that the performance of some early MedVLP methods can be enhanced to surpass more recent ones, prompting a revisiting of the developments and conclusions from prior works in MedVLP. Our code are available at https://github.com/yangzhou12/BenchX.
Contrastive learning is a form of self-supervision that can leverage unlabeled data to produce pretrained models. While contrastive learning has demonstrated promising results on natural image classification tasks, its application to medical imaging tasks like chest X-ray interpretation has been limited. In this work, we propose MoCo-CXR, which is an adaptation of the contrastive learning method Momentum Contrast (MoCo), to produce models with better representations and initializations for the detection of pathologies in chest X-rays. In detecting pleural effusion, we find that linear models t…
Concordia University（康考迪亚大学）；North South University（北南大学）
Pretraining has sparked groundswell of interest in deep learning workflows to learn from limited data and improve generalization. While this is common for 2D image classification tasks, its application to 3D medical imaging tasks like chest CT interpretation is limited. We explore the idea of whether pretraining a model on realistic videos could improve performance rather than training the model from scratch, intended for tuberculosis type classification from chest CT scans. To incorporate both spatial and temporal features, we develop a hybrid convolutional neural network (CNN) and recurrent …
RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
Department of Computer Science and Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校计算机科学与工程系）；Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）；Koita Centre for Digital Health, Indian Institute of Technology Bombay（印度理工学院孟买分校Koita数字健康中心）
Deep learning methods for chest X-ray interpretation typically rely on pretrained models developed for ImageNet. This paradigm assumes that better ImageNet architectures perform better on chest X-ray tasks and that ImageNet-pretrained weights provide a performance boost over random initialization. In this work, we compare the transfer performance and parameter efficiency of 16 popular convolutional architectures on a large chest X-ray dataset (CheXpert) to investigate these assumptions. First, we find no relationship between ImageNet performance and CheXpert performance for both models without…
Deep learning technologies have already demonstrated a high potential to build diagnosis support systems from medical imaging data, such as Chest X-Ray images. However, the shortage of labeled data in the medical field represents one key obstacle to narrow down the performance gap with respect to applications in other image domains. In this work, we investigate the benefits of a curricular Self-Supervised Learning (SSL) pretraining scheme with respect to fully-supervised training regimes for pneumonia recognition on Chest X-Ray images of Covid-19 patients. We show that curricular SSL pretraini…
Medical report generation is a challenging task since it is time-consuming and requires expertise from experienced radiologists. The goal of medical report generation is to accurately capture and describe the image findings. Previous works pretrain their visual encoding neural networks with large datasets in different domains, which cannot learn general visual representation in the specific medical domain. In this work, we propose a medical report generation framework that uses a contrastive learning approach to pretrain the visual encoder and requires no additional meta information. In additi…
A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling
Stanford Center for Artificial Intelligence in Medicine and Imaging（斯坦福大学人工智能医学与成像中心）；Stanford University（斯坦福大学）；Department of Radiology（放射科）；Department of Computer Science（计算机科学系）；Department of Pediatrics（儿科）；Department of Biomedical Data Science（生物医学数据科学系）；Department of Medicine（医学系）
Osaka University Graduate School of Medicine（大阪大学医学研究生院）
Large numbers of labeled medical images are essential for the accurate detection of anomalies, but manual annotation is labor-intensive and time-consuming. Self-supervised learning (SSL) is a training method to learn data-specific features without manual annotation. Several SSL-based models have been employed in medical image anomaly detection. These SSL methods effectively learn representations in several field-specific images, such as natural and industrial product images. However, owing to the requirement of medical expertise, typical SSL-based models are inefficient in medical image anomal…
Wadhwani Institute for Artificial Intelligence (Wadhwani AI)（瓦德瓦尼人工智能研究所（瓦德瓦尼AI））
Chest radiograph (or Chest X-Ray, CXR) is a popular medical imaging modality that is used by radiologists across the world to diagnose heart or lung conditions. Over the last decade, Convolutional Neural Networks (CNN), have seen success in identifying pathologies in CXR images. Typically, these CNNs are pretrained on the standard ImageNet classification task, but this assumes availability of large-scale annotated datasets. In this work, we analyze the utility of pretraining on unlabeled ImageNet or Chest X-Ray (CXR) datasets using various algorithms and in multiple settings. Some findings of …
Temporal Inversion for Learning Interval Change in Chest X-Rays
Interdisciplinary Program in Bioengineering, Seoul National University Graduate School（首尔大学研究生院生物工程跨学科项目）；Integrated Major in Innovative Medical Science, Seoul National University Graduate School（首尔大学研究生院创新医学科学综合专业）；Dept. of Radiology, Seoul National University Hospital（首尔大学医院放射科）；Seoul National University College of Medicine（首尔大学医学院）；Inst. of Medical and Biological Engineering, Seoul National University Medical Research Center（首尔大学医学研究中心医学与生物工程研究所）
A Generative Foundation Model for Chest Radiography
Department of Radiation Oncology, Stanford University School of Medicine（斯坦福大学医学院放射肿瘤科）；Department of Computer Science, School of Computing and Data Science, The University of Hong Kong（香港大学计算机科学系）；Department of Population Medicine and Diagnostic Sciences, College of Veterinary Medicine, Cornell University（康奈尔大学兽医学院人口医学与诊断科学系）；Department of Radiology, The First Affiliated Hospital of Jinan University（暨南大学第一附属医院放射科）；Department of Radiology, Stanford University School of Medicine（斯坦福大学医学院放射科）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）
Recent trends in self-supervised representation learning have focused on removing inductive biases from training pipelines. However, inductive biases can be useful in settings when limited data are available or provide additional insight into the underlying data distribution. We present spatial prior attention (SPAN), a framework that takes advantage of consistent spatial and semantic structure in unlabeled image datasets to guide Vision Transformer attention. SPAN operates by regularizing attention masks from separate transformer heads to follow various priors over semantic regions. These pri…
Reconstruction of Patient-Specific Confounders in AI-based Radiologic Image Interpretation using Generative Pretraining
Else Kroener Fresenius Center for Digital Health (EKFZ)（埃尔泽·克勒纳·菲森纽斯数字健康中心（EKFZ））；Technical University Dresden（德累斯顿工业大学）；University Hospital Aachen（亚琛大学医院）；UC Berkeley（加州大学伯克利分校）；University Hospital Zurich（苏黎世大学医院）；University of Zurich（苏黎世大学）；Center for Artificial Intelligence in Medicine and Imaging (AIMI)（医学与成像人工智能中心（AIMI））；Stanford University（斯坦福大学）；University Hospital Dresden（德累斯顿大学医院）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心（NCT））
Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
Interdisciplinary Program in Bioengineering, Seoul National University Graduate School（生物工程跨学科项目，首尔国立大学研究生院）；Integrated Major in Innovative Medical Science, Seoul National University Graduate School（创新医学科学整合专业，首尔国立大学研究生院）；Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院第一附属医院放射科）；Seoul National University College of Medicine（首尔国立大学医学院）；Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital（首尔国立大学医学院放射科，首尔国立大学医院）；Institute of Medical and Biological Engineering, Seoul National University Medical Research Center（医学与生物工程研究所，首尔国立大学医学研究所以及）；Institute of Radiation Medicine, Seoul National University Medical Research Center（放射医学研究所，首尔国立大学医学研究所以及）
RadTex: Learning Efficient Radiograph Representations from Text Reports
MIT Lincoln Laboratory（麻省理工学院林肯实验室）；Massachusetts Institute of Technology（麻省理工学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）
J. J. Strossmayer University of Osijek（约西普·尤拉伊·斯特罗斯马耶奥西耶克大学）；Ghent University（根特大学）
One of the largest problems in medical image processing is the lack of annotated data. Labeling medical images often requires highly trained experts and can be a time-consuming process. In this paper, we evaluate a method of reducing the need for labeled data in medical image object detection by using self-supervised neural network pretraining. We use a dataset of chest X-ray images with bounding box labels for 13 different classes of anomalies. The networks are pretrained on a percentage of the dataset without labels and then fine-tuned on the rest of the dataset. We show that it is possible …
Site-Level Fine-Tuning with Progressive Layer Freezing: Towards Robust Prediction of Bronchopulmonary Dysplasia from Day-1 Chest Radiographs in Extremely Preterm Infants
Saarland University（萨尔兰大学）；University Hospital of Saarland（萨尔兰大学医院）；Clinical Centre Karlsruhe（卡尔斯鲁厄临床中心）；Helmholtz Institute for Pharmaceutical Research Saarland（亥姆霍兹萨尔兰药物研究所）；University Hospital of Augsburg（奥格斯堡大学医院）
Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis. This paper studies the effectiveness of self-supervised learning as a pretraining strategy for medical image classification. We conduct experiments on two distinct tasks: dermatology skin condition classification from digital camera images and multi-label chest X-ray classification, and demonstrate that self-supervised learning on ImageNet, followed by additional self-supervised learning …
Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning
Raidium；Department of Vascular and Oncological Interventional Radiology, Hôpital Européen Georges Pompidou, AP-HP（欧洲乔治·蓬皮杜医院血管与肿瘤介入放射科，AP-HP）；Faculté de Santé, Université Paris-Cité（巴黎西岱大学健康学院）；HEKA, INRIA（HEKA，法国国家信息与自动化研究所）；Imaging Department, Fondation Ophtalmologique Adolphe de Rothschild（阿道夫·罗斯柴尔德眼科基金会影像科）
Tissue-Contrastive Semi-Masked Autoencoders for Segmentation Pretraining on Chest CT
Hangzhou City University（杭州城市大学）；Southwest Hospital, Army Medical University (Third Military Medical University)（陆军军医大学（第三军医大学）西南医院）；Zhejiang University（浙江大学）
Visual Prompt Engineering for Vision Language Models in Radiology
Division of Medical Image Computing, German Cancer Research Center, Heidelberg, Germany（德国癌症研究中心医学图像计算部）；Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany（海德堡大学数学与计算机科学学院）；Medical Faculty Heidelberg, University of Heidelberg, Heidelberg, Germany（海德堡大学医学学院）
Google Health（谷歌健康）；Advanced Clinical（先进临床）；Apollo Radiology International（阿波罗国际放射学）；Center of Infectious Disease Research in Zambia（赞比亚传染病研究中心）；Sibanye Stillwater（斯班耶-斯蒂尔沃特）；Clickmedix
Tuberculosis (TB) is a top-10 cause of death worldwide. Though the WHO recommends chest radiographs (CXRs) for TB screening, the limited availability of CXR interpretation is a barrier. We trained a deep learning system (DLS) to detect active pulmonary TB using CXRs from 9 countries across Africa, Asia, and Europe, and utilized large-scale CXR pretraining, attention pooling, and noisy student semi-supervised learning. Evaluation was on (1) a combined test set spanning China, India, US, and Zambia, and (2) an independent mining population in South Africa. Given WHO targets of 90% sensitivity an…
The First Affiliated Hospital of Jinan University（暨南大学附属第一医院）；The Fifth Affiliated Hospital of Sun Yat-sen University（中山大学附属第五医院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong（香港中文大学）；Future Network of Intelligence Institute (FNii)（未来智能网络研究院）；Sun Yat-sen University（中山大学）；Jinan University（暨南大学）；Guangdong Provincial Key Laboratory of Big Data Computing（广东省大数据计算重点实验室）
Medical imaging technologies, including computed tomography (CT) or chest X-Ray (CXR), are largely employed to facilitate the diagnosis of the COVID-19. Since manual report writing is usually too time-consuming, a more intelligent auxiliary medical system that could generate medical reports automatically and immediately is urgently needed. In this article, we propose to use the medical visual language BERT (Medical-VLBERT) model to identify the abnormality on the COVID-19 scans and generate the medical report automatically based on the detected lesion regions. To produce more accurate medical …
While self-supervised pretraining has proven beneficial for many computer vision tasks, it requires expensive and lengthy computation, large amounts of data, and is sensitive to data augmentation. Prior work demonstrates that models pretrained on datasets dissimilar to their target data, such as chest X-ray models trained on ImageNet, underperform models trained from scratch. Users that lack the resources to pretrain must use existing models with lower performance. This paper explores Hierarchical PreTraining (HPT), which decreases convergence time and improves accuracy by initializing the pre…
Microsoft Health Futures UK（微软健康未来英国）；Microsoft Health Futures US（微软健康未来美国）；Mayo Clinic M.D（梅奥诊所）；Radiology AI Lab, Mayo Clinic M.D（放射学AI实验室，梅奥诊所）；Microsoft Health & Life Sciences（微软健康与生命科学）；Cambridge University Hospitals M.D（剑桥大学医院）
Technical University of Munich（慕尼黑工业大学）；Helmholtz Zentrum Munich（慕尼黑亥姆霍兹中心）；Imperial College London（伦敦帝国理工学院）
Image-text contrastive learning has proven effective for pretraining medical image models. When targeting localized downstream tasks like semantic segmentation or object detection, additional local contrastive losses that align image regions with sentences have shown promising results. We study how local contrastive losses are related to global (per-sample) contrastive losses and which effects they have on localized medical downstream tasks. Based on a theoretical comparison, we propose to remove some components of local losses and replace others by a novel distribution prior which enforces un…
Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）；ETH AI Center, ETH Zurich（苏黎世联邦理工学院人工智能中心）；International School of Medicine, Istanbul Medipol University（伊斯坦布尔梅迪波尔大学国际医学院）；Department of Electrical and Computer Engineering, Boston University（波士顿大学电气与计算机工程系）；Division of Intramural Research, National Institutes of Health（美国国立卫生研究院内部研究部）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Institute for Diagnostic and Interventional Radiology, University Hospital Zurich（苏黎世大学医院诊断与介入放射学研究所）；Department Artificial Intelligence in Biomedical Engineering, FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡工业大学生物医学工程人工智能系）
CompDiff enables fair and zero shot medical image generation across demographic intersections through compositional diffusion
Institute of Data Science, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands（数据科学研究所，科学与工程学院，马斯特里赫特大学，马斯特里赫特，荷兰）；Department of Advanced Computing Sciences, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands（先进计算科学系，科学与工程学院，马斯特里赫特大学，马斯特里赫特，荷兰）；VITO, Belgium（比利时VITO研究院）
Stanford University（斯坦福大学）；Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）
We propose a selective learning method using meta-learning and deep reinforcement learning for medical image interpretation in the setting of limited labeling resources. Our method, MedSelect, consists of a trainable deep learning selector that uses image embeddings obtained from contrastive pretraining for determining which images to label, and a non-parametric selector that uses cosine similarity to classify unseen images. We demonstrate that MedSelect learns an effective selection strategy outperforming baseline selection strategies across seen and unseen medical conditions for chest X-ray …
Deep learning-based computer-aided diagnosis is gradually deployed to review and analyze medical images. However, this paradigm is restricted in real-world clinical applications due to the poor robustness and generalization. The issue is more sinister with a lack of training data. In this paper, we address the challenge from the transfer learning point of view. Different from the common setting that transferring knowledge from the natural image domain to the medical image domain, we find the knowledge from the same domain further boosts the model robustness and generalization. Therefore, we pr…
