arXiv 2407.057582024-07-08eess.IVcs.AIcs.CV

多模态大语言模型在医学图像与自由文本报告数据挖掘中的潜力

Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports

Institute of Medical Research, Northwestern Polytechnical University（西北工业大学医学研究院）；School of Computing, University of Georgia（佐治亚大学计算学院）；School of Life Science and Technology, University of Electronic Science and Technology of China（电子科技大学生命科学与技术学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Electronics and Information, Northwestern Polytechnical University（西北工业大学电子信息学院）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Lingang Laboratory（临港实验室）

Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zhengliang Liu, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究评估了Gemini、GPT-4等多模态大模型在14个医学影像及3个放射报告数据集上的多任务表现，发现两类模型各有优劣，为其临床应用的改进方向提供参考。

摘要：

医学图像与放射学报告对疾病诊断至关重要，凸显了定量分析在临床决策中的重要性。然而，这些数据的多样性与跨源异质性对当前数据挖掘方法的泛化性构成了挑战。多模态大语言模型（MLLMs）近来已变革诸多领域，对医学领域产生了显著影响。值得注意的是，Gemini-Vision系列（Gemini）与GPT-4系列（GPT-4）模型成为通用人工智能（AGI）在计算机视觉领域范式转变的典型代表，展现出其在生物医学领域的潜力。本研究中，我们评估了Gemini、GPT-4及4种主流大模型的性能，在14个医学影像数据集（涵盖5类医学影像范畴：皮肤病学、放射学、牙科学、眼科学与内镜检查）以及3个放射学报告数据集上开展了全面评估。研究涉及的任务包括疾病分类、病灶分割、解剖定位、疾病诊断、报告生成与病灶检测。实验结果表明，Gemini系列模型在报告生成与病灶检测任务中表现优异，但在疾病分类与解剖定位任务中存在不足。与之相对，GPT系列模型在病灶分割与解剖定位任务中表现出色，但在疾病诊断与病灶检测任务中遇到困难。此外，Gemini系列与GPT系列中均有模型展现出可观的生成效率。尽管两类模型在减轻医师工作负担、缓解有限医疗资源压力以及促进临床从业者与人工智能技术协作方面颇具前景，但在临床部署前仍需大幅改进与全面验证。

英文摘要：

Medical images and radiology reports are crucial for diagnosing medical conditions, highlighting the importance of quantitative analysis for clinical decision-making. However, the diversity and cross-source heterogeneity of these data challenge the generalizability of current data-mining methods. Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence (AGI) for computer vision, showcasing their potential in the biomedical domain. In this study, we evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets, including 5 medical imaging categories (dermatology, radiology, dentistry, ophthalmology, and endoscopy), and 3 radiology report datasets. The investigated tasks encompass disease classification, lesion segmentation, anatomical localization, disease diagnosis, report generation, and lesion detection. Our experimental results demonstrated that Gemini-series models excelled in report generation and lesion detection but faces challenges in disease classification and anatomical localization. Conversely, GPT-series models exhibited proficiency in lesion segmentation and anatomical localization but encountered difficulties in disease diagnosis and lesion detection. Additionally, both the Gemini series and GPT series contain models that have demonstrated commendable generation efficiency. While both models hold promise in reducing physician workload, alleviating pressure on limited healthcare resources, and fostering collaboration between clinical practitioners and artificial intelligence technologies, substantial enhancements and comprehensive validations remain imperative before clinical deployment.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。