arXiv 2406.131732024-07-16cs.CVcs.AIcs.CLcs.LG

生物医学视觉指令调优与临床医生偏好对齐

Biomedical Visual Instruction Tuning with Clinician Preference Alignment

Stanford University（斯坦福大学）；Emory University（埃默里大学）；University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）

Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, Carl Yang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出BioMed-VITAL框架，通过临床医生偏好对齐生成和选择指令数据，显著提升生物医学多模态模型的视觉对话和医学VQA性能。

摘要：

多模态基础模型的最新进展展示了在理解和推理视觉与文本信息方面的强大能力。将这些为通用用途训练的基础模型适应到生物医学等专业领域，需要大规模领域特定的指令数据集。虽然现有工作已探索自动策划此类数据集，但所得数据集并未明确与领域专业知识对齐。在本工作中，我们提出了一种以数据为中心的框架，即生物医学视觉指令调优与临床医生偏好对齐（BioMed-VITAL），该框架将临床医生的偏好纳入生成和选择用于调优生物医学多模态基础模型的指令数据的两个阶段。首先，在生成阶段，我们提示GPT-4V生成器使用一组多样化的临床医生选择的示范，用于偏好对齐的数据候选生成。然后，在选择阶段，我们训练一个独立的选择模型，该模型明确地将临床医生和政策引导的模型偏好提炼为评分函数，以选择高质量数据进行医学指令调优。结果表明，使用我们方法生成的指令遵循数据调优的模型在开放视觉对话（相对提升18.5%）和医学视觉问答（胜率高达81.73%）方面表现出显著改进。我们的指令遵循数据和模型可在BioMed-VITAL.github.io获取。

英文摘要：

Recent advancements in multimodal foundation models have showcased impressive capabilities in understanding and reasoning with visual and textual information. Adapting these foundation models trained for general usage to specialized domains like biomedicine requires large-scale domain-specific instruction datasets. While existing works have explored curating such datasets automatically, the resultant datasets are not explicitly aligned with domain expertise. In this work, we propose a data-centric framework, Biomedical Visual Instruction Tuning with Clinician Preference Alignment (BioMed-VITAL), that incorporates clinician preferences into both stages of generating and selecting instruction data for tuning biomedical multimodal foundation models. First, during the generation stage, we prompt the GPT-4V generator with a diverse set of clinician-selected demonstrations for preference-aligned data candidate generation. Then, during the selection phase, we train a separate selection model, which explicitly distills clinician and policy-guided model preferences into a rating function to select high-quality data for medical instruction tuning. Results show that the model tuned with the instruction-following data from our method demonstrates a significant improvement in open visual chat (18.5% relatively) and medical VQA (win rate up to 81.73%). Our instruction-following data and models are available at BioMed-VITAL.github.io.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。