Can LLMs' Tuning Methods Work in Medical Multimodal Domain?
Academy for Engineering and Technology, Fudan University（复旦大学工程技术学院）；Cognition and Intelligent Technology Laboratory (CIT Lab), Institute of Meta-Medical, Fudan University（复旦大学元医学研究院认知与智能技术实验室）；Engineering Research Center of AI and Robotics, Ministry of Education, China（中国教育部人工智能与机器人工程研究中心）；Jilin Provincial Key Laboratory of Intelligence Science and Engineering（吉林省智能科学与工程重点实验室）
While Large Language Models (LLMs) excel in world knowledge understanding, adapting them to specific subfields requires precise adjustments. Due to the model's vast scale, traditional global fine-tuning methods for large models can be computationally expensive and impact generalization. To address this challenge, a range of innovative Parameters-Efficient Fine-Tuning (PEFT) methods have emerged and achieved remarkable success in both LLMs and Large Vision-Language Models (LVLMs). In the medical domain, fine-tuning a medical Vision-Language Pretrained (VLP) model is essential for adapting it to specific tasks. Can the fine-tuning methods for large models be transferred to the medical field to enhance transfer learning efficiency? In this paper, we delve into the fine-tuning methods of LLMs and conduct extensive experiments to investigate the impact of fine-tuning methods for large models on the existing multimodal model in the medical domain from the training data level and the model structure level. We show the different impacts of fine-tuning methods for large models on medical VLMs and develop the most efficient ways to fine-tune medical VLP models. We hope this research can guide medical domain researchers in optimizing VLMs' training costs, fostering the broader application of VLMs in healthcare fields. The code and dataset have been released at https://github.com/TIMMY-CHAN/MILE.
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University（软件与微电子学院，北京大学）；School of Computer Science, Peking University（计算机学院，北京大学）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心，北京大学）；Peking University Sixth Hospital（北京大学第六医院）；Augusta University（奥古斯塔大学）；Peking University First Hospital（北京大学第一医院）
GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Nanjing University（南京大学）；East China Normal University（华东师范大学）；Fudan University（复旦大学）；Xiamen University（厦门大学）；Monash University（莫纳什大学）；University of Washington（华盛顿大学）；University of Cambridge（剑桥大学）；Stanford University（斯坦福大学）
针对通用 AI 缺乏医学知识的问题，构建 GMAI-VL-5.5M 多模态医学数据集并训练 GMAI-VL 模型，通过三阶段训练整合视觉文本信息，在医学视觉问答和图像诊断等任务上达到最先进性能。
MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models
Department of Electrical and Computer Engineering, University of Southern California（电气与计算机工程系，南加州大学）；Department of Radiology and Imaging Sciences, University of Utah（放射学与影像科学系，犹他大学）
MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation
Centre for Perceptual and Interactive Intelligence, the Chinese University of Hong Kong（香港中文大学感知与交互智能中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Department of Electronic Engineering, the Chinese University of Hong Kong（香港中文大学电子工程系）
MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis
The First Affiliated Hospital with Nanjing Medical University（南京医科大学第一附属医院）；Jiangsu Province Hospital（江苏省医院）；Huazhong University of Science Technology（华中科技大学）；Fudan University（复旦大学）
Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Zhihui Medical Technology (Shanghai) Co., Ltd.（智汇医疗科技（上海）有限公司）
HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）
ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue
PAII Inc.（PAII公司）；Ping An Technology（平安科技）；Ping An Healthcare and Technology Company Limited（平安医疗健康科技有限公司）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tianjin University（天津大学）
MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models
Yale University（耶鲁大学）；Korea University（韩国大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；The University of Queensland（昆士兰大学）；The University of Texas Health Science Center at Houston（德克萨斯大学休斯顿健康科学中心）；University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis
College of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能学院）；College of Artificial Intelligence, Nankai University（南开大学人工智能学院）
GraphMMP: A Graph Neural Network Model with Mutual Information and Global Fusion for Multimodal Medical Prognosis
Hangzhou Dianzi University（杭州电子科技大学）；Shenzhen Polytechnic University（深圳职业技术大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；Yunnan University（云南大学）；Shenzhen University（深圳大学）
Early Intervention for VFM-based Multimodal Medical Image Classification
Renmin University of China（中国人民大学）；Beijing Key Laboratory for Intelligent Diagnosis of Fundus Diseases and Drug-Device R&D and Translation（眼底疾病智能诊断与药械研发转化北京市重点实验室）
本文提出EI框架，通过早期干预提升多模态影像疾病识别性能，引入MoR方法优化Vision Foundation Models适应，验证了在三个公开数据集上的有效性。
Seeking Necessary and Sufficient Information from Multimodal Medical Data
University College London, London, UK（伦敦大学学院）；Imperial College London, London, UK（伦敦帝国学院）；Mingdu Tech, China（明都科技）；University of Bristol, Bristol, UK（布里斯托大学）
Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model
Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；College of AI, Tsinghua University（清华大学人工智能学院）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
ProbMed: A Probabilistic Framework for Medical Multimodal Binding
Peter Munk Cardiac Centre（彼得·默克心脏中心）；Ted Rogers Centre for Heart Research（泰德·罗杰斯心脏病研究中心）；University Health Network（大学健康网络）；Joint Department of Medical Imaging（联合医学影像部门）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis
Shanghai Artificial Intelligence Laboratory；Shanghai Innovation Institute；Shanghai Jiao Tong University；Shanghai Institute of Optics；Fudan University；University of Cambridge；Monash University；DAMO Academy, Alibaba Group；Imperial College London；The University of Hong Kong；The Hong Kong University of Science；Hupan Lab；The Chinese University of Hong Kong
