视觉-语言指令微调：综述与分析
Vision-Language Instruction Tuning: A Review and Analysis
AI总结：
本文系统综述了多模态LLMs中的视觉-语言指令微调（VLIT）设置与数据集，首次提出多视角分类并总结高质量数据特征，通过实验验证其指导作用，同时探讨了未来研究方向。
摘要：
指令微调是大型语言模型（LLMs）中关键的监督训练阶段，旨在增强LLM泛化指令执行和适应用户偏好的能力。随着多模态数据日益整合到LLMs中，对视觉-语言指令微调（VLIT）的兴趣与日俱增，与纯文本指令微调相比，它呈现出更复杂的特性。在本文中，我们系统地回顾了多模态LLMs中最新的VLIT设置及相应数据集，并深入探讨了其设计背后的内在动机。我们首次为现有VLIT数据集提供了详细的多视角分类，并明确了高质量VLIT数据应具备的特征。通过将这些特征作为指导原则纳入现有VLIT数据构建过程，我们进行了广泛实验，并验证了它们对微调后多模态LLMs性能的积极影响。此外，我们讨论了VLIT当前的挑战和未来研究方向，为该领域的持续发展提供见解。本文相关的代码和数据集已在 https://github.com/palchenli/VL-Instruction-Tuning 开源。
英文摘要：
Instruction tuning is a crucial supervised training phase in Large Language Models (LLMs), aiming to enhance the LLM's ability to generalize instruction execution and adapt to user preferences. With the increasing integration of multi-modal data into LLMs, there is growing interest in Vision-Language Instruction Tuning (VLIT), which presents more complex characteristics compared to pure text instruction tuning. In this paper, we systematically review the latest VLIT settings and corresponding datasets in multi-modal LLMs and provide insights into the intrinsic motivations behind their design. For the first time, we offer a detailed multi-perspective categorization for existing VLIT datasets and identify the characteristics that high-quality VLIT data should possess. By incorporating these characteristics as guiding principles into the existing VLIT data construction process, we conduct extensive experiments and verify their positive impact on the performance of tuned multi-modal LLMs. Furthermore, we discuss the current challenges and future research directions of VLIT, providing insights for the continuous development of this field. The code and dataset related to this paper have been open-sourced at https://github.com/palchenli/VL-Instruction-Tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
