MedPromptX：基于视觉定位的多模态提示在胸部X光诊断中的应用
MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis
AI总结：
针对胸部X光诊断中EHR不完整的问题，MedPromptX整合多模态大语言模型、少样本提示和视觉定位，补充缺失EHR信息并优化少样本数据，发布新数据集，F1分数较基线提升11%。
摘要：
胸部X光图像常用于预测急慢性心肺疾病，但由于电子健康记录（EHR）不完整，将其与结构化临床数据整合面临挑战。本文介绍MedPromptX，这是首个整合多模态大型语言模型（MLLMs）、少样本提示（FP）和视觉定位（VG）的临床决策支持系统，用于结合图像与EHR数据进行胸部X光诊断。预训练MLLM用于补充缺失的EHR信息，全面理解患者病史。此外，FP减少了MLLMs大规模训练的需求，同时有效解决幻觉问题。然而，确定最佳少样本示例数量和选择高质量候选样本的过程可能繁琐，却深刻影响模型性能。因此，我们提出一种动态优化少样本数据的新技术，以实时适应新患者场景。此外，VG缩小了X光图像中的搜索区域，从而提高异常识别能力。我们还发布了MedPromptX-VQA，这是一个新的上下文视觉问答数据集，包含来自MIMIC-IV和MIMIC-CXR-JPG数据库的交错图像和EHR数据。结果表明MedPromptX实现了SOTA性能，F1分数比基线提高11%。代码和数据可在https://github.com/BioMedIA-MBZUAI/MedPromptX公开获取。
英文摘要：
Chest X-ray images are commonly used for predicting acute and chronic cardiopulmonary conditions, but efforts to integrate them with structured clinical data face challenges due to incomplete electronic health records (EHR). This paper introduces MedPromptX, the first clinical decision support system that integrates multimodal large language models (MLLMs), few-shot prompting (FP) and visual grounding (VG) to combine imagery with EHR data for chest X-ray diagnosis. A pre-trained MLLM is utilized to complement the missing EHR information, providing a comprehensive understanding of patients' medical history. Additionally, FP reduces the necessity for extensive training of MLLMs while effectively tackling the issue of hallucination. Nevertheless, the process of determining the optimal number of few-shot examples and selecting high-quality candidates can be burdensome, yet it profoundly influences model performance. Hence, we propose a new technique that dynamically refines few-shot data for real-time adjustment to new patient scenarios. Moreover, VG narrows the search area in X-ray images, thereby enhancing the identification of abnormalities. We also release MedPromptX-VQA, a new in-context visual question answering dataset encompassing interleaved images and EHR data derived from MIMIC-IV and MIMIC-CXR-JPG databases. Results demonstrate the SOTA performance of MedPromptX, achieving an 11% improvement in F1-score compared to the baselines. Code and data are publicly available on https://github.com/BioMedIA-MBZUAI/MedPromptX.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
