arXiv 2411.113622024-11-18cs.CVcs.CL

MAIRA-Seg：通过分割感知的多模态大语言模型增强放射学报告生成

MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models

Microsoft Health Futures（微软健康未来）；Microsoft Research India（微软研究院印度分院）；University of Cambridge（剑桥大学）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院国民保健信托基金）

Harshita Sharma, Valentina Salvatelli, Shaury Srivastav, Kenza Bouzid, Shruthi Bannur, Daniel C. Castro, Maximilian Ilse, Sam Bond-Taylor, Mercy Prasanna Ranjit… 展开作者

Harshita Sharma, Valentina Salvatelli, Shaury Srivastav, Kenza Bouzid, Shruthi Bannur, Daniel C. Castro, Maximilian Ilse, Sam Bond-Taylor, Mercy Prasanna Ranjit, Fabian Falck, Fernando Pérez-García, Anton Schwaighofer, Hannah Richardson, Maria Teodora Wetscherek, Stephanie L. Hyland, Javier Alvarez-Valle

备注：: Accepted as Proceedings Paper at ML4H 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MAIRA-Seg，一个分割感知的多模态大语言模型框架，通过引入语义分割掩码和掩码感知提示，在MIMIC-CXR数据集上提升了胸部X光片放射学报告生成的性能。

摘要：

将人工智能应用于放射学报告生成，特别是胸部X光片（CXRs），正受到越来越多的关注。本文研究了通过分割掩码引入像素级信息，是否能改善多模态大语言模型（MLLMs）在放射学报告生成中的细粒度图像理解能力。我们提出了MAIRA-Seg，这是一个分割感知的MLLM框架，旨在利用语义分割掩码与CXRs一起生成放射学报告。我们训练了专家分割模型，以获取CXRs中放射学特定结构的掩码伪标签。随后，基于专门用于报告生成的CXR模型MAIRA的架构，我们集成了一个可训练的分割tokens提取器来利用这些掩码伪标签，并采用掩码感知提示来生成放射学报告草稿。我们在公开的MIMIC-CXR数据集上的实验表明，MAIRA-Seg优于非分割基线。我们还研究了MAIRA的set-of-marks提示，发现MAIRA-Seg始终表现出相当或更优的性能。结果证实，使用分割掩码增强了MLLMs的细致推理能力，可能有助于带来更好的临床结果。

英文摘要：

There is growing interest in applying AI to radiology report generation, particularly for chest X-rays (CXRs). This paper investigates whether incorporating pixel-level information through segmentation masks can improve fine-grained image interpretation of multimodal large language models (MLLMs) for radiology report generation. We introduce MAIRA-Seg, a segmentation-aware MLLM framework designed to utilize semantic segmentation masks alongside CXRs for generating radiology reports. We train expert segmentation models to obtain mask pseudolabels for radiology-specific structures in CXRs. Subsequently, building on the architectures of MAIRA, a CXR-specialised model for report generation, we integrate a trainable segmentation tokens extractor that leverages these mask pseudolabels, and employ mask-aware prompting to generate draft radiology reports. Our experiments on the publicly available MIMIC-CXR dataset show that MAIRA-Seg outperforms non-segmentation baselines. We also investigate set-of-marks prompting with MAIRA and find that MAIRA-Seg consistently demonstrates comparable or superior performance. The results confirm that using segmentation masks enhances the nuanced reasoning of MLLMs, potentially contributing to better clinical outcomes.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。