Multimodal large language models (MLLMs) represent an evolutionary expansion in the capabilities of traditional large language models, enabling them to tackle challenges that surpass the scope of purely text-based applications. It leverages the knowledge previously encoded within these language models, thereby enhancing their applicability and functionality in the reign of multimodal contexts. Recent works investigate the adaptation of MLLMs as a universal solution to address medical multi-modal problems as a generative task. In this paper, we propose a parameter efficient framework for fine-tuning MLLMs, specifically validated on medical visual question answering (Med-VQA) and medical report generation (MRG) tasks, using public benchmark datasets. We also introduce an evaluation metric using the 5-point Likert scale and its weighted average value to measure the quality of the generated reports for MRG tasks, where the scale ratings are labelled by both humans manually and the GPT-4 model. We further assess the consistency of performance metrics across traditional measures, GPT-4, and human ratings for both VQA and MRG tasks. The results indicate that semantic similarity assessments using GPT-4 align closely with human annotators and provide greater stability, yet they reveal a discrepancy when compared to conventional lexical similarity measurements. This questions the reliability of lexical similarity metrics for evaluating the performance of generative models in Med-VQA and report generation tasks. Besides, our fine-tuned model significantly outperforms GPT-4v. This indicates that without additional fine-tuning, multi-modal models like GPT-4v do not perform effectively on medical imaging tasks. The code will be available here: https://github.com/jinlHe/PeFoMed.
MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queries
Indian Institute of Technology Patna(印度理工学院巴特那分校);Stanford University(斯坦福大学);Amazon GenAI(亚马逊生成式人工智能部门);Indira Gandhi Institute of Medical Sciences(英迪拉·甘地医学科学研究所)
ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models
East China Normal University(华东师范大学);Hasso Plattner Institute/University of Potsdam(哈索·普拉特纳研究所/波茨坦大学);East China University of Science and Technology(华东理工大学)
HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale
Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));National Health Data Institute, Shenzhen(深圳国家健康数据研究院)
Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI
Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education East China University of Science and Technology(能源化工过程智能制造重点实验室,东华大学);Research Institute of Intelligent Control and Systems Harbin Institute of Technology(智能控制与系统研究室,哈尔滨工业大学);Department of Emergency Medicine, Sir Run Run Shaw Hospital Zhejiang University School of Medicine(浙江大学医学院急诊医学科);Provincial Key Laboratory of Precise Diagnosis Treatment of Abdominal Infection, Sir Run Run Shaw Hospital Zhejiang University School of Medicine(腹部感染精准诊断治疗省级重点实验室,浙江大学医学院);School of Medicine Shaoxing University(绍兴大学医学院)
Doctor Sun是一种双语多模态大语言模型,通过整合预训练视觉编码器和医学LLM,提升生物医学多模态任务的性能,并提供SunMed-VL数据集支持研究进展。
Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);Zhihui Medical Technology (Shanghai) Co., Ltd.(智汇医疗科技(上海)有限公司)
Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignment
Bosch Center for Artificial Intelligence(博世人工智能中心);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Saarland University(萨尔大学);University of Strasbourg(斯特拉斯堡大学);IHU Strasbourg(斯特拉斯堡大学医院研究所);CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心)
A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis
The University of Sydney(悉尼大学);Tencent AI Lab, Tencent(腾讯AI实验室);Guangzhou University of Chinese Medicine(广州中医药大学);The University of Wollongong(伍伦贡大学);The University of Adelaide(阿德莱德大学)
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute(伦斯勒理工学院);Wake Forest University School of Medicine(维克森林大学医学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
EmoMed: An Emotionally-Aware Agent for Multimodal Medical Support with Real-Time Information Retrieval
ISP RAS Research Center for Trusted Artificial Intelligence(俄罗斯科学院信息学与自动化研究所可信人工智能研究中心);National University of Science and Technology (NUST) MISIS(国立科技大学MISIS);Innopolis University(Innopolis大学);SkolTech(斯科尔科沃科技学院);SB AI Lab(SB人工智能实验室)
Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions
School of Biomedical Engineering, Tsinghua University(清华大学生物医学工程学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Hepato-Pancreato-Biliary Center, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua University(清华大学临床医学院北京清华长庚医院肝胆胰中心);School of Computer and Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);College of Electronic Information Engineering, Tongji University(同济大学电子与信息工程学院)
MedPrune: Topology-Efficient Multimodal Multi-Agent Communication Evolution for Medical VQA Tasks
Hefei University of Technology(合肥工业大学);Nanjing University(南京大学);Guangdong University of Finance and Economics(广东财经大学);East China Normal University(华东师范大学);Lenovo(联想集团)