Modern Vision-Language Models (VLMs) exhibit unprecedented capabilities in cross-modal semantic understanding between visual and textual modalities. Given the intrinsic need for multi-modal integration in clinical applications, VLMs have emerged as a promising solution for a wide range of medical image analysis tasks. However, adapting general-purpose VLMs to medical domain poses numerous challenges, such as large domain gaps, complicated pathological variations, and diversity and uniqueness of different tasks. The central purpose of this review is to systematically summarize recent advances in adapting VLMs for medical image analysis, analyzing current challenges, and recommending promising yet urgent directions for further investigations. We begin by introducing core learning strategies for medical VLMs, including pretraining, fine-tuning, and prompt learning. We then categorize five major VLM adaptation strategies for medical image analysis. These strategies are further analyzed across eleven medical imaging tasks to illustrate their current practical implementations. Furthermore, we analyze key challenges that impede the effective adaptation of VLMs to clinical applications and discuss potential directions for future research. We also provide an open-access repository of related literature to facilitate further research, available at https://github.com/haonenglin/Awesome-VLM-for-MIA. It is anticipated that this article can help researchers who are interested in harnessing VLMs in medical image analysis tasks have a better understanding on their capabilities and limitations, as well as current technical barriers, to promote their innovative, robust, and safe application in clinical practice.
E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；iFlytek Co.Ltd（科大讯飞股份有限公司）；The First Affiliated of USTC, Anhui Provincial Hospital（中国科学技术大学附属第一医院（安徽省立医院））
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University（软件与微电子学院，北京大学）；School of Computer Science, Peking University（计算机学院，北京大学）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心，北京大学）；Peking University Sixth Hospital（北京大学第六医院）；Augusta University（奥古斯塔大学）；Peking University First Hospital（北京大学第一医院）
A Survey of Medical Vision-and-Language Applications and Their Techniques
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；School of Software Engineering（软件工程学院）；South China University of Technology（华南理工大学）；South Australia Medical Imaging（南澳大利亚医学影像中心）；School of Computer Science & Engineering（计算机科学与工程学院）；Northwestern Polytechnical University（西北工业大学）
RelativeFlow: Taming Medical Image Denoising Learning with Noisy Reference
School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院，中国）；Department of Biomedical Engineering, Case Western Reserve University, USA（凯斯西储大学生物医学工程系，美国）；School of Instrument Science and Engineering, Southeast University, China（东南大学仪器科学与工程学院，中国）；Department of Optical Sciences, University of Arizona, USA（亚利桑那大学光学科学系，美国）
UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Models
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University（集成航空航天地面海洋大数据应用技术国家工程实验室，计算机科学与工程学院，西北工业大学）
West China Hospital, Sichuan University（四川大学华西医院）；Sichuan University（四川大学）；BUPT（北京邮电大学）；Shanghai AI-Lab（上海人工智能实验室）
The large-scale pre-trained vision language models (VLM) have shown remarkable domain transfer capability on natural images. However, it remains unknown whether this capability can also apply to the medical image domain. This paper thoroughly studies the knowledge transferability of pre-trained VLMs to the medical domain, where we show that well-designed medical prompts are the key to elicit knowledge from pre-trained VLMs. We demonstrate that by prompting with expressive attributes that are shared between domains, the VLM can carry the knowledge across domains and improve its generalization. …
Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To solve medical VQA problems with a limited number of training data, pretrain-finetune paradigm is widely used to improve the model generalization. In this paper, we propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining on medical image caption da…
When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains
University of Toronto, Canada（多伦多大学）；Vector Institute, Canada（向量研究所）；KITE Research Institute, University Health Network, Canada（KITE研究机构）；York University, Canada（约克大学）
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Eisai Inc.（卫材株式会社）；IISc, Bangalore（印度科学研究所班加罗尔分校）；Cohere Labs Community（Cohere实验室社区）
GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Nanjing University（南京大学）；East China Normal University（华东师范大学）；Fudan University（复旦大学）；Xiamen University（厦门大学）；Monash University（莫纳什大学）；University of Washington（华盛顿大学）；University of Cambridge（剑桥大学）；Stanford University（斯坦福大学）
针对通用 AI 缺乏医学知识的问题，构建 GMAI-VL-5.5M 多模态医学数据集并训练 GMAI-VL 模型，通过三阶段训练整合视觉文本信息，在医学视觉问答和图像诊断等任务上达到最先进性能。
Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
Visual Computing Group, Institute of Media Informatics, Ulm University, Germany（媒体信息研究所视觉计算组，乌尔姆大学，德国）；Diagnostic and Interventional Radiology, Ulm University Medical Center, Germany（乌尔姆大学医学中心诊断与介入放射学）；Axiom Bio, USA（Axiom Bio公司，美国）
MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Sabaragamuwa University of Sri Lanka（斯里兰卡萨巴拉加穆瓦大学）；Digital Platform Development, SLT PLC（SLT公共有限公司数字平台开发部）
RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding
Shenzhen Institutes of Advanced Technology, Chinese Academy of Science（深圳先进技术研究院，中国科学院）；CyPhi( ) AI Research Lab, School of IT, Monash University, Malaysia Campus（CyPhi人工智能研究实验室，信息学院，墨尔本大学马来西亚校区）
Scaling Down to Scale Up: Towards Operationally-Efficient and Deployable Clinical Models via Cross-Modal Low-Rank Adaptation for Medical Vision-Language Models
King Fahd University of Petroleum（国王法赫德石油与矿物大学）；Institute for Medical Engineering（医学工程研究所）；Science, Massachusetts Institute of Technology, US（科学，麻省理工学院，美国）；Harvard Medical School, Harvard University, US（哈佛医学院，哈佛大学，美国）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence, Saudi Arabia（SDAIA-KFUPM人工智能联合研究中心，沙特阿拉伯）
MedCLM: Learning to Localize and Reason via a CoT-Curriculum in Medical Vision-Language Models
A.I.MATICS Inc（A.I.MATICS公司）；Boston University（波士顿大学）；University of Southern California（南加州大学）；Heuron（Heuron公司）；MODULABS, Open Neural Networks Research Lab（MODULABS，开放神经网络研究实验室）
HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Alibaba（阿里巴巴）；The Hong Kong University of Science and Technology（香港科技大学）；National University of Singapore（新加坡国立大学）
ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue
PAII Inc.（PAII公司）；Ping An Technology（平安科技）；Ping An Healthcare and Technology Company Limited（平安医疗健康科技有限公司）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tianjin University（天津大学）
HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）
EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models
Jilin University（吉林大学）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；ByteDance（字节跳动）；Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）
Benchmarking Direct Preference Optimization for Medical Large Vision-Language Models
Korea University（韩国大学）；AIGEN Sciences（AIGEN公司）；Hanyang University College of Medicine（翰林大学医学院）；Asan Medical Center, University of Ulsan College of Medicine（釜山大学医学院阿桑医院）；Yale University（耶鲁大学）
