面向医学视觉定位的参数高效微调医学多模态大语言模型
Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
AI总结：
针对医学MLLMs训练成本高、数据需求大及视觉定位任务输出格式适配难的问题，提出参数高效微调模型PFMVG，在公开医学视觉定位基准上表现优于GPT-4v。
摘要：
多模态大语言模型（MLLMs）继承了大语言模型（LLMs）卓越的文本理解能力，并将这些能力拓展至多模态场景，在通用领域的多模态任务中取得了优异成果。然而在医学领域，高昂的训练成本和对大量医学数据的需求给医学MLLMs的发展带来了挑战。此外，由于答案采用自由文本形式，视觉定位等需要按规定格式输出的任务对MLLMs来说难度较大。截至目前，医学视觉定位领域尚无相关医学MLLMs研究。针对需根据短文本描述识别医学图像中位置的医学视觉定位任务，我们提出了面向医学视觉定位的参数高效微调医学多模态大语言模型（PFMVG）。为验证模型性能，我们在一个公开的医学视觉定位基准数据集上对其进行评估，模型取得了具有竞争力的结果，且性能显著优于GPT-4v。我们的代码将在同行评审后开源。
英文摘要：
Multimodal Large Language Models (MLLMs) inherit the superior text understanding capabilities of LLMs and extend these capabilities to multimodal scenarios. These models achieve excellent results in the general domain of multimodal tasks. However, in the medical domain, the substantial training costs and the requirement for extensive medical data pose challenges to the development of medical MLLMs. Furthermore, due to the free-text form of answers, tasks such as visual grounding that need to produce output in a prescribed form become difficult for MLLMs. So far, there have been no medical MLLMs works in medical visual grounding area. For the medical vision grounding task, which involves identifying locations in medical images based on short text descriptions, we propose Parameter-efficient Fine-tuning medical multimodal large language models for Medcial Visual Grounding (PFMVG). To validate the performance of the model, we evaluate it on a public benchmark dataset for medical visual grounding, where it achieves competitive results, and significantly outperforming GPT-4v. Our code will be open sourced after peer review.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
