Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classification
Clinical Medical Research Center, The First People’s Hospital of Kashi (Kashgar) Prefecture（喀什地区第一人民医院临床医学研究中心）；Xinjiang Key Laboratory of Artificial Intelligence Assisted Imaging Diagnosis（新疆人工智能辅助影像诊断重点实验室）；Department of Hepatobiliary Surgery, The First People’s Hospital of Kashi (Kashgar) Prefecture（喀什地区第一人民医院肝胆外科）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
X-ray imaging is pivotal in medical diagnostics, offering non-invasive insights into a range of health conditions. Recently, vision-language models, such as the Contrastive Language-Image Pretraining (CLIP) model, have demonstrated potential in improving diagnostic accuracy by leveraging large-scale image-text datasets. However, since CLIP was not initially designed for medical images, several CLIP-like models trained specifically on medical images have been developed. Despite their enhanced performance, issues of fairness - particularly regarding demographic attributes - remain largely unaddressed. In this study, we perform a comprehensive fairness analysis of CLIP-like models applied to X-ray image classification. We assess their performance and fairness across diverse patient demographics and disease categories using zero-shot inference and various fine-tuning techniques, including Linear Probing, Multilayer Perceptron (MLP), Low-Rank Adaptation (LoRA), and full fine-tuning. Our results indicate that while fine-tuning improves model accuracy, fairness concerns persist, highlighting the need for further fairness interventions in these foundational models.
Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine（放射肿瘤科，Winship癌症研究所，埃默里大学医学院）；Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology（生物医学工程系，工程学院，佐治亚理工学院）；Department of Computer Science and Mathematics, Laney Graduate School, Emory University（计算机科学与数学系，兰尼研究生院，埃默里大学）；School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology（电气与计算机工程系，工程学院，佐治亚理工学院）
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Department of Computer Science and Engineering（计算机科学与工程系）；Incheon National University（庆尚国立大学）；Department of Artificial Intelligence（人工智能系）；Inha University（Inha大学）；Center for Brain-Machine Interface（脑机接口中心）
Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong（香港中文大学医学智能与扩展现实研究所）
CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology
Westlake University（西湖大学）；Zhejiang University（浙江大学）；Harvard University（哈佛大学）；The Ohio State University（俄亥俄州立大学）；University of the Chinese Academy of Sciences（中国科学院大学）
Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment
organization= Orion Lab, National Observatory of Athens \& National Technical University of Athens , country= Greece；organization= Department of Informatics \& Telematics, Harokopio University of Athens , country= Greece；organization= Faculty of Electrical Engineering；organization= BIFOLD - Berlin Institute for the Foundations of Learning
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Foundation models trained on large-scale dataset gain a recent surge in CV and NLP. In contrast, development in biomedical domain lags far behind due to data scarcity. To address this issue, we build and release PMC-OA, a biomedical dataset with 1.6M image-caption pairs collected from PubMedCentral's OpenAccess subset, which is 8 times larger than before. PMC-OA covers diverse modalities or diseases, with majority of the image-caption samples aligned at finer-grained level, i.e., subfigure and subcaption. While pretraining a CLIP-style model on PMC-OA, our model named PMC-CLIP achieves state-o…
To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Computer Science and Engineering, Texas A&M University（德克萨斯农工大学计算机科学与工程系）；Computer Science and Engineering, Georgia Institute of Technology（佐治亚理工学院计算机科学与工程系）
DGTRSD & DGTRS-CLIP: A Dual-Granularity Remote Sensing Image-Text Dataset and Vision Language Foundation Model for Alignment
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院 aerospace information research institute）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Information Network Security, People’s Public Security University of China（中国人民公安大学信息网络安全学院）
RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports
Beijing Institute of Technology（北京理工大学）；Beijing Tongren Hospital（北京同仁医院）；Capital Medical University（首都医科大学）；Beijing Institute of Ophthalmology（北京眼科研究所）
An Empirical Study on the Fairness of Foundation Models for Multi-Organ Image Segmentation
Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布莱根妇女医院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）；MR Business Unit, Philips Healthcare（飞利浦医疗磁共振事业部）；Simens Healthineers Ltd., China（西门子医疗中国有限公司）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学研究中心）
FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models
The University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（矢量研究所）；University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong（香港中文大学）
The University of Tokyo（东京大学）；Google Research（谷歌研究院）
Domain generalization (DG) is a difficult transfer learning problem aiming to learn a generalizable model for unseen domains. Recent foundation models (FMs) are robust to many distribution shifts and, therefore, should substantially improve the performance of DG. In this work, we study generic ways to adopt CLIP, a Visual-Language Foundation Model, for DG problems in image classification. While ERM greatly improves the accuracy with bigger backbones and training datasets using standard DG benchmarks, fine-tuning FMs is not practical in many real-world situations. We propose Domain Prompt Learn…
University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；University of Cambridge（剑桥大学）
The goal of this work is to segment and name regions of images without access to pixel-level labels during training. To tackle this task, we construct segmenters by distilling the complementary strengths of two foundation models. The first, CLIP (Radford et al. 2021), exhibits the ability to assign names to image content but lacks an accessible representation of object structure. The second, DINO (Caron et al. 2021), captures the spatial extent of objects but has no knowledge of object names. Our method, termed NamedMask, begins by using CLIP to construct category-specific archives of images. …
DeepCORO-CLIP: A Multi-View Foundation Model for Comprehensive Coronary Angiography Video-Text Analysis and External Validation
Division of Cardiology, Department of Medicine, Montreal Heart Institute（蒙特利尔心脏研究所心血管科，医学部）；McGill, Faculty of Engineering（麦吉尔大学工程学院）；Division of Cardiology, Department of Medicine, University of California, San Francisco（加州大学旧金山分校医学部心血管科）；Cardiovascular Research Institute, University of California, San Francisco（加州大学旧金山分校心血管研究所）；Bakar Computational Health Sciences Institute, University of California, San Francisco（加州大学旧金山分校Bakar计算健康科学研究所）；Department of Cardiology, Smidt Heart Institute, Cedars-Sinai Medical Center, Los Angeles, CA（Cedars-Sinai 医疗中心洛杉矶分院心血管科，Smidt心脏研究所）；Department of Bioengineering, University of California Los Angeles（加州大学洛杉矶分校生物工程系）；Department of Medicine, Kaiser Permanente, San Francisco, USA（Kaiser Permanente 旧金山分院医学部）；Department of Medicine, Université de Montréal（蒙特利尔大学医学部）
Cardiac-CLIP: A Vision-Language Foundation Model for 3D Cardiac CT Images
Southeast University（东南大学）；Jinling Hospital, Affiliated Hospital of Medical School, Nanjing University（南京大学医学院附属金陵医院）；The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院）；Shanghai AI Laboratory（上海人工智能实验室）；Nanjing First Hospital, Nanjing Medical University（南京医科大学附属南京医院）；Northern Jiangsu People’s Hospital（苏北人民医院）；the Affiliated Hospital of Jiangnan University（江南大学附属医院）；Sir Run Run Shaw Hospital, Zhejiang University School of Medicine（浙江大学医学院附属邵逸夫医院）；Beijing Anzhen Hospital, Capital Medical University（首都医科大学附属北京安贞医院）；Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属第一人民医院）；Guangdong Provincial People’s Hospital（广东省人民医院）；Xijing Hospital, Air Force Medical University（空军军医大学西京医院）；Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院北京协和医院）；Sir Run Run Hospital, Nanjing Medical University（南京医科大学附属逸夫医院）；Taizhou People’s Hospital（泰州市人民医院）；Case Western Reserve University（凯斯西储大学）
CT-CLIP Representations for Multimodal Lung Cancer Survival Prediction
Chalmers University of Technology（查尔姆斯理工大学）；Umeå University（于默奥大学）；Sahlgrenska University Hospital（萨尔格伦斯卡大学医院）；Sahlgrenska Academy at Gothenburg University（哥德堡大学萨尔格伦斯卡学院）
