arXiv 2402.116902024-02-18cs.CLcs.CV

Vision-Flan：在视觉指令微调中扩展人工标注任务

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning

Virginia Tech（弗吉尼亚理工大学）；University of Michigan（密歇根大学）；University of Washington（华盛顿大学）；Amazon Inc.（亚马逊公司）；Meta AI；Microsoft（微软公司）

Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang

备注：: 8 Pages, visual instruction tuning

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对VLM任务多样性不足及GPT-4合成数据存在偏差的问题，构建含187个任务的Vision-Flan数据集，并提出两阶段指令微调框架，显著提升多模态性能，发现少量GPT-4数据即可对齐人类偏好。

摘要：

尽管视觉语言模型（VLMs）作为多功能视觉助手展现出卓越能力，但现有VLM框架中仍存在两个重大挑战：（1）预训练和视觉指令微调中缺乏任务多样性；（2）GPT-4合成的指令微调数据中存在标注错误和偏差。这两个挑战导致了泛化性差、幻觉和灾难性遗忘等问题。为了解决这些挑战，我们构建了Vision-Flan，这是迄今为止最多样化的公开视觉指令微调数据集，包含来自学术数据集的187个多样化任务和1,664,261个实例，每个任务都配有专家编写的指令。此外，我们提出了一个两阶段指令微调框架，其中VLMs首先在Vision-Flan上进行微调，然后在GPT-4合成数据上进一步微调。我们发现这种两阶段微调框架显著优于传统的单阶段视觉指令微调框架，并在广泛的多模态评估基准上实现了最先进的性能。最后，我们进行了深入分析以理解视觉指令微调，我们的发现揭示：（1）GPT-4合成数据并未实质性增强VLMs的能力，而是调节模型响应以符合人类偏好的格式；（2）极少数量（例如1,000）的GPT-4合成数据就能有效将VLM响应与人类偏好对齐；（3）视觉指令微调主要帮助大型语言模型（LLMs）理解视觉特征。

英文摘要：

Despite vision-language models' (VLMs) remarkable capabilities as versatile visual assistants, two substantial challenges persist within the existing VLM frameworks: (1) lacking task diversity in pretraining and visual instruction tuning, and (2) annotation error and bias in GPT-4 synthesized instruction tuning data. Both challenges lead to issues such as poor generalizability, hallucination, and catastrophic forgetting. To address these challenges, we construct Vision-Flan, the most diverse publicly available visual instruction tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances sourced from academic datasets, and each task is accompanied by an expert-written instruction. In addition, we propose a two-stage instruction tuning framework, in which VLMs are firstly finetuned on Vision-Flan and further tuned on GPT-4 synthesized data. We find this two-stage tuning framework significantly outperforms the traditional single-stage visual instruction tuning framework and achieves the state-of-the-art performance across a wide range of multi-modal evaluation benchmarks. Finally, we conduct in-depth analyses to understand visual instruction tuning and our findings reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs' capabilities but rather modulates the model's responses to human-preferred formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can effectively align VLM responses with human-preference; (3) Visual instruction tuning mainly helps large-language models (LLMs) to understand visual features.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。