通过创建与LLM对齐的指令弥合视觉指令微调中的写作风格差距
Bridging Writing Manner Gap in Visual Instruction Tuning by Creating LLM-aligned Instructions
AI总结：
该研究针对视觉指令与基础大语言模型间的写作风格差距导致模型能力退化的问题，提出用基础LLM对齐软格式视觉指令的写作风格，生成对齐指令，有效缩小差距并提升基线模型的抗幻觉能力与多基准综合性能。
摘要：
在大型多模态模型（Large Multi-modal Models, LMMs）领域，视觉指令微调阶段的指令质量显著影响模态对齐的性能。本文从一个被称为“写作风格”的独特视角评估指令质量，写作风格涵盖为传达特定语义所采用的词汇选择、语法和句式结构。我们认为，视觉指令与LMM内部的基础大语言模型（Large Language Models, LLMs）之间存在显著的写作风格差距。这一差距迫使预训练基础LLM偏离其原有写作风格，导致基础LLM和LMM的能力均出现退化。为在保留原始语义的同时弥合写作风格差距，我们提出直接利用基础LLM，将软格式视觉指令的写作风格与基础LLM自身的写作风格对齐，由此生成全新的LLM-aligned指令（与大语言模型对齐的指令）。人工写作风格评估结果表明，我们的方法成功缩小了写作风格差距。通过使用LLM-aligned指令，基线模型LLaVA-7B和QwenVL展现出更强的抗幻觉能力，并且在全部15个视觉与语言基准测试中取得了显著的综合性能提升。
英文摘要：
In the realm of Large Multi-modal Models (LMMs), the instruction quality during the visual instruction tuning stage significantly influences the performance of modality alignment. In this paper, we assess the instruction quality from a unique perspective termed \textbf{Writing Manner}, which encompasses the selection of vocabulary, grammar and sentence structure to convey specific semantics. We argue that there exists a substantial writing manner gap between the visual instructions and the base Large Language Models (LLMs) within LMMs. This gap forces the pre-trained base LLMs to deviate from their original writing styles, leading to capability degradation of both base LLMs and LMMs. To bridge the writing manner gap while preserving the original semantics, we propose directly leveraging the base LLM to align the writing manner of soft-format visual instructions with that of the base LLM itself, resulting in novel LLM-aligned instructions. The manual writing manner evaluation results demonstrate that our approach successfully minimizes the writing manner gap. By utilizing LLM-aligned instructions, the baseline models LLaVA-7B and QwenVL demonstrate enhanced resistance to hallucinations and non-trivial comprehensive improvements across all $15$ visual and language benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
