用于医学报告生成的多模态大语言模型：基于定制化提示微调
Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuning
AI总结：
本文提出MRG-LLM，一种结合冻结大语言模型与可学习视觉编码器的多模态模型，通过动态定制提示机制生成实例特定提示，在IU X-ray和MIMIC-CXR数据集上实现医学报告生成的最先进性能。
摘要：
从影像数据生成医学报告在临床实践中仍是一项具有挑战性的任务。尽管大语言模型（LLMs）在解决这一挑战方面展现出巨大潜力，但其与医学影像数据的有效整合仍值得深入探索。在本文中，我们提出了MRG-LLM，一种新颖的多模态大语言模型（MLLM），它将冻结的LLM与可学习的视觉编码器相结合，并引入了一种动态提示定制机制。我们的关键创新在于通过从视觉特征导出的条件仿射变换，为每张个体医学图像生成实例特定的提示。我们提出了两种实现方式：基于提示的定制和基于提示库的定制，从而实现精准且有目标性的报告生成。在IU X-ray和MIMIC-CXR数据集上进行的大量实验表明，MRG-LLM在医学报告生成方面达到了最先进的性能。我们的代码将公开提供。
英文摘要：
Medical report generation from imaging data remains a challenging task in clinical practice. While large language models (LLMs) show great promise in addressing this challenge, their effective integration with medical imaging data still deserves in-depth exploration. In this paper, we present MRG-LLM, a novel multimodal large language model (MLLM) that combines a frozen LLM with a learnable visual encoder and introduces a dynamic prompt customization mechanism. Our key innovation lies in generating instance-specific prompts tailored to individual medical images through conditional affine transformations derived from visual features. We propose two implementations: prompt-wise and promptbook-wise customization, enabling precise and targeted report generation. Extensive experiments on IU X-ray and MIMIC-CXR datasets demonstrate that MRG-LLM achieves state-of-the-art performance in medical report generation. Our code will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
