Align$^2$LLaVA: 多模态指令编纂的级联人类与大语言模型偏好对齐
Align$^2$LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation
AI总结：
Align$^2$LLaVA通过级联人类与LLM偏好对齐方法，有效压缩多模态指令数据，提升模型性能。
摘要：
近年来，多模态大语言模型（MLLMs）如LLaVA系列模型的进步主要由大量机器生成的指令跟随数据训练驱动。然而，此类自动指令收集流程却无意中引入了数据质量的显著变化。本文提出了一种新的指令编纂算法，从两个独特的视角出发：人类和LLM偏好对齐，以压缩庞大的机器生成多模态指令集为紧凑且高质量的形式：（i）对于人类偏好对齐，我们收集了一个机器生成的多模态指令数据集，并建立了一套全面的主观和客观标准，以指导人类专家对数据质量进行关键评估。通过这样做，我们训练了一个奖励模型在注释数据集上，以内化人类对指令对齐的细微理解。（ii）对于LLM偏好对齐，给定奖励模型所选的指令，我们提出利用MLLM中使用的内部LLM来对视觉指令的写作风格与自身内部LLM进行对齐，从而实现LLM对齐的指令改进。大量实验表明，我们可以通过压缩合成多模态指令最多90%来维持或甚至提高模型性能。令人印象深刻的是，通过将训练指令从158k大幅减少到14k（9倍更小），我们的模型在各种MLLM基准测试中始终优于其全尺寸数据集的对应物。我们的项目可在https://github.com/DCDmllm/Align2LLaVA上获得。
英文摘要：
Recent advances in Multi-modal Large Language Models (MLLMs), such as LLaVA-series models, are driven by massive machine-generated instruction-following data tuning. Such automatic instruction collection pipelines, however, inadvertently introduce significant variability in data quality. This paper introduces a novel instruction curation algorithm, derived from two unique perspectives, human and LLM preference alignment, to compress this vast corpus of machine-generated multimodal instructions to a compact and high-quality form: (i) For human preference alignment, we have collected a machine-generated multimodal instruction dataset and established a comprehensive set of both subjective and objective criteria to guide the data quality assessment critically from human experts. By doing so, a reward model was trained on the annotated dataset to internalize the nuanced human understanding of instruction alignment. (ii) For LLM preference alignment, given the instruction selected by the reward model, we propose leveraging the inner LLM used in MLLM to align the writing style of visual instructions with that of the inner LLM itself, resulting in LLM-aligned instruction improvement. Extensive experiments demonstrate that we can maintain or even improve model performance by compressing synthetic multimodal instructions by up to 90%. Impressively, by aggressively reducing the training instructions from 158k to 14k (9$\times$ smaller), our model consistently outperforms its full-size dataset counterpart across various MLLM benchmarks. Our project is available at https://github.com/DCDmllm/Align2LLaVA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
