arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.02797cs.CLcs.AI
收藏

PeFoMed:面向医学影像的多模态大语言模型参数高效微调

PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging

Harbin Engineering University(哈尔滨工程大学);Monash University(莫纳什大学);National Imaging Facility(国家成像设施)

Jinlong He, Pengfei Li, Gang Liu, Genrong He, Zhaolin Chen, Shenjun Zhong

AI总结:

本研究提出面向医学影像的MLLMs参数高效微调框架PeFoMed,在Med-VQA和MRG任务上验证其性能优于GPT-4v,并指出传统词汇相似性指标评估生成式医学模型的可靠性存疑。

摘要:

多模态大语言模型(MLLMs)是传统大语言模型能力的演进式拓展,使其能够应对超出纯文本应用范畴的挑战。它利用此前编码在这些语言模型中的知识,从而提升其在多模态语境领域的适用性与功能性。近期研究探索将MLLMs作为通用方案,以生成式任务的形式解决医学多模态问题。本文提出一种用于微调MLLMs的参数高效框架,并使用公开基准数据集,在医学视觉问答(Med-VQA)和医学报告生成(MRG)任务上进行了专项验证。我们还引入了一种采用5点李克特量表及其加权平均值的评估指标,用于衡量MRG任务中生成报告的质量,该量表的评分由人工标注和GPT-4模型共同完成。我们进一步评估了VQA和MRG任务中,传统衡量指标、GPT-4评分与人工评分之间的性能一致性。结果表明,基于GPT-4的语义相似性评估与人工标注者的结果高度吻合,且稳定性更强,但与传统的词汇相似性衡量方法相比存在差异。这对词汇相似性指标在评估Med-VQA和报告生成任务中生成式模型性能的可靠性提出了质疑。此外,我们的微调模型性能显著优于GPT-4v。这表明,在未经过额外微调的情况下,类似GPT-4v的多模态模型无法在医学影像任务上有效发挥作用。代码将发布于:https://github.com/jinlHe/PeFoMed。

英文摘要:

Multimodal large language models (MLLMs) represent an evolutionary expansion in the capabilities of traditional large language models, enabling them to tackle challenges that surpass the scope of purely text-based applications. It leverages the knowledge previously encoded within these language models, thereby enhancing their applicability and functionality in the reign of multimodal contexts. Recent works investigate the adaptation of MLLMs as a universal solution to address medical multi-modal problems as a generative task. In this paper, we propose a parameter efficient framework for fine-tuning MLLMs, specifically validated on medical visual question answering (Med-VQA) and medical report generation (MRG) tasks, using public benchmark datasets. We also introduce an evaluation metric using the 5-point Likert scale and its weighted average value to measure the quality of the generated reports for MRG tasks, where the scale ratings are labelled by both humans manually and the GPT-4 model. We further assess the consistency of performance metrics across traditional measures, GPT-4, and human ratings for both VQA and MRG tasks. The results indicate that semantic similarity assessments using GPT-4 align closely with human annotators and provide greater stability, yet they reveal a discrepancy when compared to conventional lexical similarity measurements. This questions the reliability of lexical similarity metrics for evaluating the performance of generative models in Med-VQA and report generation tasks. Besides, our fine-tuned model significantly outperforms GPT-4v. This indicates that without additional fine-tuning, multi-modal models like GPT-4v do not perform effectively on medical imaging tasks. The code will be available here: https://github.com/jinlHe/PeFoMed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑