arXiv 2406.006672024-06-02eess.IVcs.AIcs.CLcs.CVcs.LG

多模态大语言模型在医学成像中应用的早期探索

An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging

Faculty of Computing and Information Technology, Sohar University（苏哈尔大学计算与信息技术学院）；Foundation University School of Sciences and Technology（基础大学科学与技术学院）；College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）

Sulaiman Khan, Md. Rafiul Biswas, Alina Murad, Hazrat Ali, Zubair Shah

备注：: Accepted in Fifth IEEE Workshop on Artificial Intelligence for HealthCare, IEEE 25th International Conference on Information Reuse and Integration for Data Science

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究探索了Gemini和GPT-4V在医学图像分析中的潜力，发现Gemini在真实与合成图像分类任务上略优于GPT-4V，揭示了多模态大语言模型的潜力与局限。

摘要：

多模态大语言模型（MLLMs）的最新进展激发了人们对其在各种医学成像领域潜在应用的浓厚兴趣。一方面，人们试图使用这些生成模型来合成逼真的医学图像数据，另一方面，在数据池中识别合成图像数据的能力也至关重要。在本研究中，我们使用两种模态的医学图像数据，探索了Gemini（gemini-1.0-pro-vision-latest）和GPT-4V（gpt-4-vision-preview）模型在医学图像分析中的潜力。利用合成和真实成像数据，Gemini AI和GPT-4V首先被用于对真实图像与合成图像进行分类，随后对输入图像进行解释和分析。实验结果表明，Gemini和GPT-4都能对输入图像进行一定的解释。在此特定实验中，Gemini在分类任务上的表现略优于GPT-4V。相比之下，GPT-4V的响应大多具有通用性质。我们这项早期调查提供了关于MLLMs辅助视网膜眼底镜和肺部X光图像分类与解释的潜力的见解。我们还指出了针对医学图像分析专业任务的MLLMs早期调查研究中存在的主要局限性。

英文摘要：

Recent developments in multimodal large language models (MLLMs) have spurred significant interest in their potential applications across various medical imaging domains. On the one hand, there is a temptation to use these generative models to synthesize realistic-looking medical image data, while on the other hand, the ability to identify synthetic image data in a pool of data is also significantly important. In this study, we explore the potential of the Gemini (\textit{gemini-1.0-pro-vision-latest}) and GPT-4V (gpt-4-vision-preview) models for medical image analysis using two modalities of medical image data. Utilizing synthetic and real imaging data, both Gemini AI and GPT-4V are first used to classify real versus synthetic images, followed by an interpretation and analysis of the input images. Experimental results demonstrate that both Gemini and GPT-4 could perform some interpretation of the input images. In this specific experiment, Gemini was able to perform slightly better than the GPT-4V on the classification task. In contrast, responses associated with GPT-4V were mostly generic in nature. Our early investigation presented in this work provides insights into the potential of MLLMs to assist with the classification and interpretation of retinal fundoscopy and lung X-ray images. We also identify key limitations associated with the early investigation study on MLLMs for specialized tasks in medical image analysis.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。