arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.07971cs.CV
收藏

XrayGPT:使用医学视觉-语言模型进行胸片摘要生成

XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models

Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Aalto University(艾尔沃斯大学)

Omkar Thawakar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, Fahad Shahbaz Khan

AI总结:

本文提出XrayGPT,通过将医学视觉编码器MedClip与微调后的语言模型Vicuna线性对齐,并利用约217k条放射学报告摘要进行微调,实现胸片的开放式视觉问答与自动摘要。

摘要:

大型视觉-语言模型(如Bard和GPT-4)的最新突破展示了其在执行广泛任务方面的非凡能力。这类模型在海量数据集上训练,这些数据集包含数十亿公共图像-文本对并涵盖多种任务。然而,它们在特定任务领域(如放射学)上的表现仍未得到充分研究,并且由于对生物医学图像理解能力的不足,其潜力可能受到限制。另一方面,对话式医学模型已取得显著成功,但主要集中在基于文本的分析上。在本文中,我们提出XrayGPT,一种新颖的对话式医学视觉-语言模型,能够分析并回答关于胸片的开放式问题。具体而言,我们通过简单的线性变换将医学视觉编码器(MedClip)与微调后的大型语言模型(Vicuna)对齐。这种对齐使我们的模型具备卓越的视觉对话能力,并建立在对放射影像和医学领域知识的深入理解之上。为提升大型语言模型在医学语境中的性能,我们从自由文本放射学报告中生成了约217k条交互式高质量摘要。这些摘要通过微调过程增强了大型语言模型的性能。我们的方法为推进胸片自动分析研究开辟了新途径。我们的开源演示、模型和指令集可在以下地址获取:https://github.com/mbzuai-oryx/XrayGPT。

英文摘要:

The latest breakthroughs in large vision-language models, such as Bard and GPT-4, have showcased extraordinary abilities in performing a wide range of tasks. Such models are trained on massive datasets comprising billions of public image-text pairs with diverse tasks. However, their performance on task-specific domains, such as radiology, is still under-investigated and potentially limited due to a lack of sophistication in understanding biomedical images. On the other hand, conversational medical models have exhibited remarkable success but have mainly focused on text-based analysis. In this paper, we introduce XrayGPT, a novel conversational medical vision-language model that can analyze and answer open-ended questions about chest radiographs. Specifically, we align both medical visual encoder (MedClip) with a fine-tuned large language model (Vicuna), using a simple linear transformation. This alignment enables our model to possess exceptional visual conversation abilities, grounded in a deep understanding of radiographs and medical domain knowledge. To enhance the performance of LLMs in the medical context, we generate ~217k interactive and high-quality summaries from free-text radiology reports. These summaries serve to enhance the performance of LLMs through the fine-tuning process. Our approach opens up new avenues the research for advancing the automated analysis of chest radiographs. Our open-source demos, models, and instruction sets are available at: https://github.com/mbzuai-oryx/XrayGPT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑