LLaVA-Med：在一天内训练一个面向生物医学的大型语言与视觉助手
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
AI总结：
本文提出LLaVA-Med，通过PubMed Central图表-说明数据、GPT-4自指令数据和课程学习微调通用视觉-语言模型，在15小时内训练出生物医学多模态对话助手，并在部分VQA指标上达到最优。
摘要：
对话式生成式AI在赋能生物医学从业者方面已展现出显著前景，但当前研究主要集中于单模态文本。多模态对话式AI通过利用来自公共网络的数十亿图文对取得了快速进展，但此类通用领域视觉-语言模型在理解生物医学图像并围绕其开展对话方面仍不够成熟。本文提出一种具有成本效益的方法，用于训练视觉-语言对话助手，使其能够回答有关生物医学图像的开放式研究问题。核心思路是利用从PubMed Central提取的大规模、覆盖广泛的生物医学图表-说明数据集，使用GPT-4根据说明自指令生成开放式指令遵循数据，然后采用一种新颖的课程学习方法微调大型通用领域视觉-语言模型。具体而言，模型首先使用原始图表-说明对学习对齐生物医学词汇，随后使用GPT-4生成的指令遵循数据学习掌握开放式对话语义，大致模拟外行逐步获取生物医学知识的过程。这使我们能够在不到15小时内（使用8块A100）训练出一个生物医学大型语言与视觉助手（Large Language and Vision Assistant for BioMedicine，LLaVA-Med）。LLaVA-Med展现出出色的多模态对话能力，能够遵循开放式指令，协助处理有关生物医学图像的查询。在三个标准生物医学视觉问答数据集上，LLaVA-Med在某些指标上优于以往的监督式最先进方法。为促进生物医学多模态研究，我们将发布指令遵循数据和LLaVA-Med模型。
英文摘要：
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
