arXiv 2506.000602025-05-29cs.CYcs.AIcs.CL

关于隐私保护的开源LLM在从临床CMR影像报告中提取诊断信息的比较分析

Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports

The D-Lab, Department of Precision Medicine, GROW - Research Institute for Oncology and Reproduction, Maastricht University（D实验室，精准医学系，GROW癌症与生殖研究研究所，马斯特里赫特大学）；Division of Cardiovascular Imaging, Department of Cardiology I, University Hospital Münster（心血管影像科，心内科第一部门，穆恩斯特大学医院）

Sina Amirrajab, Volker Vehof, Michael Bietenbeck, Ali Yilmaz

备注：: under review for Scientific Reports

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究评估了9个隐私保护的开源LLMs从109份临床CMR报告中提取诊断信息的能力，发现排名前四的模型在所有指标上均超越委员会认证心脏病专家，证实了开源LLMs在临床影像报告自动分析中的可行性。

摘要：

目的：我们研究了利用隐私保护、本地部署的开源大型语言模型（LLMs）从自由文本的心血管磁共振（CMR）报告中提取诊断信息。材料与方法：我们评估了9个开源LLMs，基于109份临床CMR报告中的描述性发现，评估其识别诊断并将患者分类至各种心脏诊断类别的能力。性能通过标准分类指标进行量化，包括准确率、精确率、召回率和F1分数。我们还采用混淆矩阵来检查各模型间的误分类模式。结果：大多数开源LLMs在将报告分类至不同诊断类别时表现出色。Google的Gemma2模型获得了0.98的最高平均F1分数，其次是Qwen2.5:32B和DeepseekR1-32B，F1分数分别为0.96和0.95。所有其他评估模型的平均得分均在0.93以上，Mistral和DeepseekR1-7B是仅有的例外。在分析CMR报告时，排名前四的LLMs在所有评估指标上均超越了我们的委员会认证心脏病专家（F1分数为0.94）。结论：我们的发现证明了在临床环境中实施开源、隐私保护的LLMs用于影像报告自动分析的可行性，实现了准确、快速且资源高效的诊断分类。

英文摘要：

Purpose: We investigated the utilization of privacy-preserving, locally-deployed, open-source Large Language Models (LLMs) to extract diagnostic information from free-text cardiovascular magnetic resonance (CMR) reports. Materials and Methods: We evaluated nine open-source LLMs on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptive findings in 109 clinical CMR reports. Performance was quantified using standard classification metrics including accuracy, precision, recall, and F1 score. We also employed confusion matrices to examine patterns of misclassification across models. Results: Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories. Google's Gemma2 model achieved the highest average F1 score of 0.98, followed by Qwen2.5:32B and DeepseekR1-32B with F1 scores of 0.96 and 0.95, respectively. All other evaluated models attained average scores above 0.93, with Mistral and DeepseekR1-7B being the only exceptions. The top four LLMs outperformed our board-certified cardiologist (F1 score of 0.94) across all evaluation metrics in analyzing CMR reports. Conclusion: Our findings demonstrate the feasibility of implementing open-source, privacy-preserving LLMs in clinical settings for automated analysis of imaging reports, enabling accurate, fast and resource-efficient diagnostic categorization.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。