StableLLaVA：利用合成图像-对话数据增强视觉指令微调
StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
AI总结：
提出StableLLaVA方法，利用ChatGPT和文本到图像生成模型同步合成图像-对话数据用于视觉指令微调，缓解领域偏差，在多个多模态基准上取得最优结果。
摘要：
OpenAI的GPT-4所展现出的卓越多模态能力，激发了人们对多模态大语言模型（LLMs）开发的浓厚兴趣。这类模型的一个主要研究目标是，在有效理解人类指令的同时，对齐视觉和文本模态。当前的方法通常依赖于从基准数据集中获取的标注来构建用于训练的图像-对话数据集，类似于LLM中的指令微调。然而，这些数据集往往表现出领域偏差，可能限制模型的生成能力。为了缓解这些局限性，我们提出了一种新颖的数据收集方法，该方法同步合成用于视觉指令微调的图像和对话。这种方法利用了生成模型的力量，将ChatGPT和文本到图像生成模型的能力相结合，以生成具有多样化图像内容的、多样且可控的数据集。此外，数据集可以任意扩展。这不仅比现有方法提供了更大的灵活性，还显著增强了模型的多种能力。我们的研究包括在各种数据集上进行的全面实验。结果强调了在十多种常见评估能力上的显著提升。此外，我们的模型在多个广泛认可的多模态基准上取得了最先进的结果。
英文摘要：
The remarkable multimodal capabilities demonstrated by OpenAI's GPT-4 have sparked significant interest in the development of multimodal Large Language Models (LLMs). A primary research objective of such models is to align visual and textual modalities effectively while comprehending human instructions. Current methodologies often rely on annotations derived from benchmark datasets to construct image-dialogue datasets for training purposes, akin to instruction tuning in LLMs. However, these datasets often exhibit domain bias, potentially constraining the generative capabilities of the models. In an effort to mitigate these limitations, we propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models to yield a diverse and controllable dataset with varied image content. Additionally, datasets can be arbitrarily scaled. This not only provides greater flexibility compared to existing methodologies but also significantly enhances several model capabilities. Our research includes comprehensive experiments conducted on various datasets. The results emphasize substantial enhancements in more than ten commonly assessed capabilities. Additionally, our model achieves state-of-the-art results across multiple widely recognized multimodal benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
