arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11262cs.CV
收藏

生成式视觉指令微调

Generative Visual Instruction Tuning

Rice University(莱斯大学);Google DeepMind(谷歌DeepMind)

Jefferson Hernandez, Ruben Villegas, Vicente Ordonez

AI总结:

本文提出 GenLLaVA,通过结合 Mistral、SigLIP 与 StableDiffusion 三类预训练模型并利用 GPT-4V 构建的指令数据进行微调,在提升视觉理解的同时支持图像生成与编辑。

摘要:

我们提出利用自动生成的指令遵循数据来提升大型多模态模型的零样本能力,并额外支持生成式与图像编辑任务。我们通过使用 GPT-4V 以及现有的图像生成和编辑数据集,构建了一个新的多模态指令遵循数据集。利用该指令集和现有的用于视觉理解任务的 LLaVA-Finetune 指令集,我们构建了 GenLLaVA,一个生成式大语言与视觉助手。GenLLaVA 通过一种策略构建,该策略通过指令微调将三类大型预训练模型结合起来:用于语言建模的 Mistral、用于图像-文本匹配的 SigLIP,以及用于文本到图像生成的 StableDiffusion。我们的模型展现出优于 LLaVA 的视觉理解能力,并在与 Unified-IO 2 等原生多模态模型的比较中展现出具有竞争力的结果,为通过有效复用现有多模态模型构建先进通用视觉助手铺平了道路。我们开源了数据集、代码库和模型检查点,以促进该领域的进一步研究和应用。

英文摘要:

We propose to use automatically generated instruction-following data to improve the zero-shot capabilities of a large multimodal model with additional support for generative and image editing tasks. We achieve this by curating a new multimodal instruction-following set using GPT-4V and existing datasets for image generation and editing. Using this instruction set and the existing LLaVA-Finetune instruction set for visual understanding tasks, we produce GenLLaVA, a Generative Large Language and Visual Assistant. GenLLaVA is built through a strategy that combines three types of large pretrained models through instruction finetuning: Mistral for language modeling, SigLIP for image-text matching, and StableDiffusion for text-to-image generation. Our model demonstrates visual understanding capabilities superior to LLaVA and additionally demonstrates competitive results with native multimodal models such as Unified-IO 2, paving the way for building advanced general-purpose visual assistants by effectively re-using existing multimodal models. We open-source our dataset, codebase, and model checkpoints to foster further research and application in this domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑