大视觉-语言模型的指令遵循评估
Instruction-Following Evaluation of Large Vision-Language Models
AI总结：
本研究评估了大视觉-语言模型在微调后的指令遵循能力，发现其能力下降并分析了原因，提出通过包含输出格式指示的训练数据集可缓解此问题。
摘要：
在大规模语言模型（LLMs）初步繁荣之后，出现了大量整合语言能力与视觉能力的大视觉-语言模型（LVLMs）。然而，观察到在使用常用训练数据集对LVLMs进行微调后，它们往往无法表现出在整合前存在的指令遵循能力，导致结果不符合预期。本研究定量证明了在微调后LVLMs的指令遵循能力下降，并分析了其根本原因。特别是，我们构建了新的训练数据集，突出输出格式是否被指定。然后，我们研究了在微调过程中明确指示输出格式如何影响LVLMs的指令遵循能力。定量评估确认，使用常用数据集微调后，LVLMs的指令遵循能力下降。此外，我们发现训练数据集包含输出格式指示的LVLMs比不包含的模型更准确地遵循指令。这些发现表明，在（视觉）指令微调过程中包含输出格式指示的样本可能有助于缓解指令遵循能力的下降。
英文摘要：
Following the initial flourishing of large language models (LLMs), there has been a surge in proposed large vision-language models (LVLMs) that integrate LLMs with vision capabilities. However, it has been observed that LVLMs, after tuning to visual instruction using commonly used training datasets, often fail to exhibit the instruction-following ability that was present in the LLM before integration, leading to results in which they do not follow task instructions as expected. This study quantitatively demonstrates that LVLMs' instruction-following ability declines after fine-tuning and analyzes its underlying causes. In particular, we constructed new training datasets highlighting whether the output format is specified. Then, we investigated how explicitly indicating the output format during fine-tuning affects LVLMs' instruction-following ability. Our quantitative evaluation confirmed that LVLMs' instruction-following ability declines after fine-tuning with commonly used datasets. Furthermore, we found that LVLMs trained with datasets, including instructions on output format, tend to follow instructions more accurately than models that do not. These findings suggest that including samples with instructions on output format during (visual) instruction tuning may help mitigate the decline in instruction-following abilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
