迈向多模态大语言模型的鲁棒指令微调
Towards Robust Instruction Tuning on Multimodal Large Language Models
AI总结：
本文提出自动指令增强方法 INSTRAUG，从少量元指令扩展多模态指令数据 30 倍，并在 MULTIINSTRUCT 与 InstructBLIP 上验证其可显著提升 MLLM 的跨任务对齐效果。
摘要：
在多任务指令遵循数据上微调大语言模型（LLM），已被证明是提升其在新任务上零样本能力的一种强大学习范式。近期关于高质量指令遵循数据生成与选择的工作需要大量人力，以便为给定任务构思模型可理解的指令，并仔细筛选 LLM 生成的数据。在本工作中，我们提出一种用于多模态任务的自动指令增强方法，名为 INSTRAUG。该方法从少量基础且直接的元指令出发，却能将指令遵循数据集扩展 30 倍。在两个流行的多模态指令遵循基准 MULTIINSTRUCT 和 InstructBLIP 上的结果表明，INSTRAUG 能够显著改善多模态大语言模型（MLLM）在 12 项多模态任务中的对齐效果，其收益甚至等同于将训练数据规模扩大数倍所带来的收益。
英文摘要：
Fine-tuning large language models (LLMs) on multi-task instruction-following data has been proven to be a powerful learning paradigm for improving their zero-shot capabilities on new tasks. Recent works about high-quality instruction-following data generation and selection require amounts of human labor to conceive model-understandable instructions for the given tasks and carefully filter the LLM-generated data. In this work, we introduce an automatic instruction augmentation method named INSTRAUG in multimodal tasks. It starts from a handful of basic and straightforward meta instructions but can expand an instruction-following dataset by 30 times. Results on two popular multimodal instructionfollowing benchmarks MULTIINSTRUCT and InstructBLIP show that INSTRAUG can significantly improve the alignment of multimodal large language models (MLLMs) across 12 multimodal tasks, which is even equivalent to the benefits of scaling up training data multiple times.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
