面向视觉语言模型的多语言训练和评估资源
Multilingual Training and Evaluation Resources for Vision-Language Models
AI总结：
本文提出跨五种欧洲语言的视觉语言模型训练与评估资源，通过再生与翻译方法生成高质量多语言数据，验证多语言数据在非英语基准上的有效性。
摘要：
视觉语言模型（VLMs）近年来取得了快速进展。然而，尽管其发展依赖于英语，导致两个主要限制：（i）缺乏多语言和多模态数据集用于训练，（ii）缺乏跨语言的全面评估基准。本文通过引入覆盖五种欧洲语言（英语、法语、德语、意大利语和西班牙语）的新型综合资源来填补这些空白。我们采用再生-翻译范式，通过结合精心挑选的合成生成和人工标注来生成高质量的跨语言资源。具体而言，我们构建了Multi-PixMo训练语料库，通过再生Pixmo现有数据集中的示例，结合许可的模型：PixMo-Cap、PixMo-AskModelAnything和CoSyn-400k。在评估方面，我们构建了一组多语言基准，通过翻译广泛使用的英语数据集（MMbench、ScienceQA、MME、POPE、AI2D）来实现。我们通过定性和定量的人类分析评估这些资源的质量，测量跨标注者的一致性。此外，我们进行了消融研究，以展示多语言数据在VLMs训练中的影响，相对于仅英语数据。实验包括三种不同的模型，结果表明使用多语言、多模态示例训练VLMs在非英语基准上始终有益，同时对英语也有积极的迁移效果。
英文摘要：
Vision Language Models (VLMs) achieved rapid progress in the recent years. However, despite their growth, VLMs development is heavily grounded on English, leading to two main limitations: (i) the lack of multilingual and multimodal datasets for training, and (ii) the scarcity of comprehensive evaluation benchmarks across languages. In this work, we address these gaps by introducing a new comprehensive suite of resources for VLMs training and evaluation spanning five European languages (English, French, German, Italian, and Spanish). We adopt a regeneration-translation paradigm that produces high-quality cross-lingual resources by combining curated synthetic generation and manual annotation. Specifically, we build Multi-PixMo, a training corpus obtained regenerating examples from Pixmo pre-existing datasets with permissively licensed models: PixMo-Cap, PixMo-AskModelAnything, and CoSyn-400k. On the evaluation side, we construct a set of multilingual benchmarks derived translating widely used English datasets (MMbench, ScienceQA, MME, POPE, AI2D). We assess the quality of these resources through qualitative and quantitative human analyses, measuring inter-annotator agreement. Additionally, we perform ablation studies to demonstrate the impact of multilingual data, with respect to English only, in VLMs training. Experiments, comprising 3 different models show that using multilingual, multimodal examples for training VLMs aids is consistently beneficial on non-English benchmarks, with positive transfer to English as well.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
