arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.11490cs.CVcs.AIcs.CLcs.LG
收藏

LLM-CXR:面向胸部X光图像理解与生成的指令微调大语言模型

LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation

Korea Advanced Institute of Science & Technology(韩国科学技术院)

Suhyeon Lee, Won Jun Kim, Jinho Chang, Jong Chul Ye

AI总结:

本文提出LLM-CXR,通过指令微调仅文本预训练的LLM,使其获得胸部X光图像的理解与生成能力,在更小模型规模下实现更好的图像-文本对齐。

摘要:

随着大语言模型(LLM)的显著发展,LLM中的视觉-语言对齐研究正积极推进,以实现多模态推理和视觉输入输出。这一研究方向与医学影像尤为相关,因为医学图像分析和生成涉及基于视觉特征与先验知识相结合的逻辑推理。近期许多工作聚焦于训练适配器网络,作为图像处理网络与LLM之间的信息桥梁;但推测而言,为了充分发挥LLM在视觉信息上的推理潜力,视觉和语言特征应被允许更自由地交互。这在医学领域尤为重要,因为理解与生成胸部X光(CXR)等医学图像不仅需要准确的视觉和基于语言的推理,还需要两种模态之间更紧密的映射。因此,借鉴先前关于Transformer与VQ-GAN结合实现双向图像和文本生成的工作,我们在此方法基础上进行扩展,开发了一种对仅经过文本预训练的LLM进行指令微调的方法,使其获得针对医学图像的视觉-语言能力。具体而言,我们利用预训练LLM现有的问答和指令遵循能力,通过指示其回答关于图像输入的问题来教导其理解视觉输入,并对称地,通过使用涵盖基于图像的文本生成和基于文本的图像生成的多样化任务对LLM进行微调,使其能针对给定查询输出适当的文本和图像响应。我们展示了以这种方式训练的模型LLM-CXR,在CXR理解和生成任务中均表现出更好的图像-文本对齐,同时其规模小于先前开发的执行范围较窄任务的模型。代码位于https://github.com/hyn2028/llm-cxr。

英文摘要:

Following the impressive development of LLMs, vision-language alignment in LLMs is actively being researched to enable multimodal reasoning and visual IO. This direction of research is particularly relevant to medical imaging because medical image analysis and generation consist of reasoning based on a combination of visual features and prior knowledge. Many recent works have focused on training adapter networks that serve as an information bridge between image processing networks and LLMs; but presumably, in order to achieve maximum reasoning potential of LLMs on visual information as well, visual and language features should be allowed to interact more freely. This is especially important in the medical domain because understanding and generating medical images such as chest X-rays (CXR) require not only accurate visual and language-based reasoning but also a more intimate mapping between the two modalities. Thus, taking inspiration from previous work on the transformer and VQ-GAN combination for bidirectional image and text generation, we build upon this approach and develop a method for instruction-tuning an LLM pre-trained only on text to gain vision-language capabilities for medical images. Specifically, we leverage a pretrained LLM's existing question-answering and instruction-following abilities to teach it to understand visual inputs by instructing it to answer questions about image inputs and, symmetrically, output both text and image responses appropriate to a given query by tuning the LLM with diverse tasks that encompass image-based text-generation and text-based image-generation. We show that our model, LLM-CXR, trained in this approach shows better image-text alignment in both CXR understanding and generation tasks while being smaller in size compared to previously developed models that perform a narrower range of tasks. The code is at https://github.com/hyn2028/llm-cxr.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑