mBLIP: 多语种视觉大语言模型的高效引导构建
mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs
AI总结：
本研究提出高效构建的多语种视觉大语言模型mBLIP，通过将适配英文LLM的图像编码器与多语种LLM重新对齐，在消费级硬件上即可训练，性能比肩SOTA且远超仅英文的同类模型。
摘要：
模块化视觉语言模型（Vision-LLMs）将预训练图像编码器与（冻结的）大语言模型（LLMs）对齐，并对LLMs进行后设条件处理以使其“理解”图像输入。由于现成的高质量英文图文数据丰富，且单语英文LLMs性能强劲，当前研究重点一直集中在仅支持英文的Vision-LLMs上。多语种视觉语言模型仍主要通过成本高昂的端到端预训练获得，导致模型规模相对较小，且训练所用的多语种图像数据有限，仅辅以纯文本多语种语料库。我们提出mBLIP，这是首个利用多语种LLMs的Vision-LLM，我们在消费级硬件上以计算高效的方式训练得到该模型。为此，我们使用仅数百万条多语种训练样本，将此前针对英文LLM调优的图像编码器与全新的多语种LLM“重新对齐”；这些样本来自多种视觉语言任务的混合数据，是通过将高质量英文数据机器翻译为95种语言得到的。在IGLUE基准和XM3600数据集上，mBLIP取得了与当前最优模型相当的结果，且性能大幅优于Llava 1.5等强劲的仅英文Vision-LLMs。我们在https://github.com/gregor-ge/mBLIP 发布了模型、代码和训练数据。
英文摘要：
Modular vision-language models (Vision-LLMs) align pretrained image encoders with (frozen) large language models (LLMs) and post-hoc condition LLMs to `understand' the image input. With the abundance of readily available high-quality English image-text data as well as strong monolingual English LLMs, the research focus has been on English-only Vision-LLMs. Multilingual vision-language models are still predominantly obtained via expensive end-to-end pretraining, resulting in comparatively smaller models, trained on limited multilingual image data supplemented with text-only multilingual corpora. We present mBLIP, the first Vision-LLM leveraging multilingual LLMs, which we obtain in a computationally efficient manner on consumer-level hardware. To this end, we \textit{re-align} an image encoder previously tuned to an English LLM to a new, multilingual LLM using only a few million multilingual training examples derived from a mix of vision-and-language tasks, which we obtain by machine-translating high-quality English data to 95 languages. On the IGLUE benchmark and XM3600, mBLIP yields results competitive with state-of-the-art models and it greatly outperforms strong English-only Vision-LLMs like Llava 1.5. We release our model, code, and train data at \url{https://github.com/gregor-ge/mBLIP}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
