arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.05680cs.AIcs.CLcs.CV
收藏

多模态大语言模型对CT扫描的解读能力如何?一种用于分析的自动评估框架

How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses

National Center for Biotechnology Information, National Library of Medicine(国家生物技术信息中心、美国国家医学图书馆);Clinical Center(临床中心);Imperial College London(伦敦帝国学院);King Abdullah University of Science & Technology(阿卜杜拉国王科技大学);Center for Information Technology, National Institutes of Health(美国国立卫生研究院信息技术中心)

Qingqing Zhu, Benjamin Hou, Tejas S. Mathai, Pritam Mukherjee, Qiao Jin, Xiuying Chen, Zhizheng Wang, Ruida Cheng, Ronald M. Summers, Zhiyong Lu

AI总结:

针对CT扫描自动解读缺乏评估数据集与标准的问题,提出GPTRadScore评估框架,通过GPT-4分解技术对比多模态大模型生成的CT描述与金标准,验证了其相关性优于传统指标,还通过微调RadFM提升了准确率验证了假设。

摘要:

自动解读CT扫描能够减轻放射科医生的工作负担。然而,这一任务颇具挑战性,主要原因在于缺乏充足的数据集和评估参考标准。本研究旨在通过提出一种名为“GPTRadScore”的新型评估框架来弥补这一差距。该框架用于评估多模态大语言模型(例如带视觉功能的GPT-4(GPT-4V)、Gemini Pro Vision、LLaVA-Med和RadFM)为前瞻性识别的病灶生成描述的能力。通过采用基于GPT-4的分解技术,GPTRadScore将这些生成的描述与金标准报告语句进行对比,从身体部位、位置和病灶类型三个维度分析其准确性。评估结果显示,该框架与临床医生的评估具有高度相关性,并且展现出优于BLEU、METEOR和ROUGE等传统指标的潜力。此外,为助力未来研究,我们计划发布一个由临床医生标注的基准数据集。借助GPTRadScore,我们发现尽管GPT-4V和Gemini Pro Vision表现更佳,但它们的性能仍存在很大的提升空间,这主要是由于这些模型训练所用数据集的局限性所致。为验证这一潜力,我们对RadFM进行了微调,其准确性得到了显著提升:位置准确率从3.41%提升至12.8%,身体部位准确率从29.12%提升至53%,病灶类型准确率从9.24%提升至30%,从而验证了我们的假设。

英文摘要:

Automatically interpreting CT scans can ease the workload of radiologists. However, this is challenging mainly due to the scarcity of adequate datasets and reference standards for evaluation. This study aims to bridge this gap by introducing a novel evaluation framework, named ``GPTRadScore''. This framework assesses the capabilities of multi-modal LLMs, such as GPT-4 with Vision (GPT-4V), Gemini Pro Vision, LLaVA-Med, and RadFM, in generating descriptions for prospectively-identified findings. By employing a decomposition technique based on GPT-4, GPTRadScore compares these generated descriptions with gold-standard report sentences, analyzing their accuracy in terms of body part, location, and type of finding. Evaluations demonstrated a high correlation with clinician assessments and highlighted its potential over traditional metrics, such as BLEU, METEOR, and ROUGE. Furthermore, to contribute to future studies, we plan to release a benchmark dataset annotated by clinicians. Using GPTRadScore, we found that while GPT-4V and Gemini Pro Vision fare better, their performance revealed significant areas for improvement, primarily due to limitations in the dataset used for training these models. To demonstrate this potential, RadFM was fine-tuned and it resulted in significant accuracy improvements: location accuracy rose from 3.41\% to 12.8\%, body part accuracy from 29.12\% to 53\%, and type accuracy from 9.24\% to 30\%, thereby validating our hypothesis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑