面向三维脑CT报告生成的多模态大语言模型整体框架
Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation
AI总结：
本研究针对三维脑CT报告生成的数据、模型与评估局限，构建3D-BrainCT数据集，通过临床视觉指令微调训练BrainGPT模型，并提出新型评估指标FORTE，显著提升了报告生成的临床相关性与类人表现。
摘要：
多模态大语言模型（MLLMs）已被广泛应用于探索令人兴奋的医疗应用，主要集中在放射学报告生成。然而，2D放射学描述的初步成功不足以反映体积3D解剖结构在现实世界中的诊断挑战。为缓解现有文献中的三个关键局限方面，包括（1）数据复杂性，（2）模型容量，和（3）评估指标保真度，我们收集了一个包含18,885对文本-扫描对的3D-BrainCT数据集，并应用临床视觉指令微调（CVIT）来训练BrainGPT模型，以生成符合放射学标准的3D脑CT报告。在统计上，我们的BrainGPT在内部测试中得分为BLEU-1 = 44.35，BLEU-4 = 20.38，METEOR = 30.13，ROUGE-L = 47.6，和CIDEr-R = 211.77，并在外部验证CQ500数据集上描述中线移位的准确率达到0.91。通过进一步检查生成的报告，我们发现传统指标似乎仅测量表面文本相似性，无法衡量诊断目的的信息密度。为弥补这一差距，我们提出了一种新的面向特征的放射学任务评估（FORTE）来估计报告的临床相关性（病灶特征和标志点）。值得注意的是，BrainGPT模型的平均FORTE F1分数为0.71（degree=0.661；landmark=0.706；feature=0.693；impression=0.779）。为证明BrainGPT模型具备生成类人放射学报告的客观准备度，我们进行了一项图灵测试，招募了11名医生评估者，约74%的BrainGPT生成的描述与人类撰写的难以区分。我们的工作体现了一个整体框架，展示了策划3D脑CT数据集、微调解剖学感知语言模型和提出稳健放射学评估指标的第一手经验。
英文摘要：
Multi-modal large language models (MLLMs) have been given free rein to explore exciting medical applications with a primary focus on radiology report generation. Nevertheless, the preliminary success in 2D radiology captioning is incompetent to reflect the real-world diagnostic challenge in the volumetric 3D anatomy. To mitigate three crucial limitation aspects in the existing literature, including (1) data complexity, (2) model capacity, and (3) evaluation metric fidelity, we collected an 18,885 text-scan pairs 3D-BrainCT dataset and applied clinical visual instruction tuning (CVIT) to train BrainGPT models to generate radiology-adherent 3D brain CT reports. Statistically, our BrainGPT scored BLEU-1 = 44.35, BLEU-4 = 20.38, METEOR = 30.13, ROUGE-L = 47.6, and CIDEr-R = 211.77 during internal testing and demonstrated an accuracy of 0.91 in captioning midline shifts on the external validation CQ500 dataset. By further inspecting the captioned report, we reported that the traditional metrics appeared to measure only the surface text similarity and failed to gauge the information density of the diagnostic purpose. To close this gap, we proposed a novel Feature-Oriented Radiology Task Evaluation (FORTE) to estimate the report's clinical relevance (lesion feature and landmarks). Notably, the BrainGPT model scored an average FORTE F1-score of 0.71 (degree=0.661; landmark=0.706; feature=0.693; impression=0.779). To demonstrate that BrainGPT models possess objective readiness to generate human-like radiology reports, we conducted a Turing test that enrolled 11 physician evaluators, and around 74% of the BrainGPT-generated captions were indistinguishable from those written by humans. Our work embodies a holistic framework that showcased the first-hand experience of curating a 3D brain CT dataset, fine-tuning anatomy-sensible language models, and proposing robust radiology evaluation metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
