arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11684cs.CLcs.AI
收藏

ALLaVA:利用 GPT4V 合成数据训练轻量级视觉-语言模型

ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models

Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, Benyou Wang

AI总结:

本文提出 ALLaVA 合成数据生成流程,利用 GPT4V 产出 130 万对齐标注与复杂推理问答样本,使轻量级 VLM 在多项基准上媲美更大规模模型。

摘要:

大型视觉-语言模型(LVLMs)凭借强大的推理和泛化能力,已在广泛的视觉-语言任务中展现出应用前景。然而,它们在训练和部署时需要大量计算资源。本研究旨在通过采用高质量训练数据,弥合传统规模 LVLMs 与资源友好的轻量版本之间的性能差距。为此,我们提出了一条用于生成合成数据集的完整流程。其核心思想是利用强大的专有模型生成:(i)用于视觉-语言对齐的细粒度图像标注,以及(ii)用于视觉指令微调的复杂推理视觉问答对,最终共得到 130 万个样本。我们在该合成数据集上训练了一系列轻量级 VLM,实验结果证明了所提方案的有效性:这些模型在 17 个基准上取得了 4B LVLM 中具有竞争力的性能,甚至在多个基准上表现与 7B/13B 规模模型相当。本工作强调了利用高质量数据构建更高效 LVLM 的可行性。我们将数据集命名为 ALLaVA,并向研究社区开源,以开发更节省资源、适用范围更广的 LVLMs。

英文摘要:

Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset \textit{ALLaVA}, and open-source it to research community for developing better resource-efficient LVLMs for wider usage.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑