MLAN：基于语言的指令微调在多模态语言模型中保留并迁移知识
MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
AI总结：
提出一种偏重文本的视觉指令微调策略，通过引入多样化纯文本数据建立知识库，在12个数据集上以一半训练token实现跨模态知识迁移并保持零样本性能。
摘要：
我们提出了一种新颖的视觉指令微调策略，通过建立坚实的纯文本知识库来提升多模态大语言模型的零样本任务泛化能力。现有工作在指令微调阶段对各模态的重要性缺乏充分实验，通常使用大量视觉-语言数据，同时限制纯文本数据并固定模态混合比例。通过在视觉指令微调阶段引入多样化的纯文本数据，我们在各种受控实验中改变视觉-语言数据，以探究模态在视觉指令微调中的重要性。我们的全面评估表明，在12个通用数据集上，这种偏重文本的指令微调方法在两种模态上的表现均与传统偏重视觉的混合方法持平，且仅使用低至一半的总训练token。我们发现，仅需增加足够多样化的纯文本数据，即可实现跨模态的指令遵循能力和领域知识迁移，且比视觉-语言方法更高效。
英文摘要：
We present a novel visual instruction tuning strategy to improve the zero-shot task generalization of multimodal large language models by building a firm text-only knowledge base. Existing work lacks sufficient experimentation on the importance of each modality in the instruction tuning stage, often using a majority of vision-language data while keeping text-only data limited and fixing mixtures of modalities. By incorporating diverse text-only data in the visual instruction tuning stage, we vary vision-language data in various controlled experiments to investigate the importance of modality in visual instruction tuning. Our comprehensive evaluation shows that the text-heavy instruction tuning approach is able to perform on-par with traditional vision-heavy mixtures on both modalities across 12 general datasets while using as low as half the total training tokens. We find that simply increasing sufficiently diverse text-only data enables transfer of instruction following ability and domain knowledge across modalities while being more efficient than the vision-language approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
