构建多语言视觉-文本数据集以揭示视觉语言模型的视觉多语言能力
Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
AI总结：
针对现有数据集无法全面评估视觉语言模型跨语言细粒度视觉语言能力的问题，本研究构建了四种语言的多语言视觉-文本数据集，提出新任务与推理依据机制，验证了数据集的适用性与微调可行性。
摘要：
大语言模型（LLMs）提升了人们对视觉语言模型（VLMs）的研究兴趣，这类模型以图文对作为输入处理。目前已有针对VLMs视觉理解能力的研究，但这类研究仍处于初步阶段，因为现有数据集无法对VLMs跨多种语言的细粒度视觉语言能力开展全面评估。为进一步挖掘GPT-4V等VLMs的优势，我们构建了新的数据集用于对VLMs进行系统的定性分析。我们的贡献分为四点：1）提出了9项视觉与语言（VL）任务（包括物体识别、图文匹配等），并通过使用包含“问题”的模板、调用GPT-4V生成“答案”和“推理依据”，构建了英语、日语、斯瓦希里语、乌尔都语四种语言的多语言视觉-文本数据集；2）提出了一项名为“无关性”的新型VL任务；3）引入推理依据以帮助人类理解VLM的推理过程；4）通过人工评估衡量所提数据集对VL任务的适用性。我们证明VLMs可以在我们的数据集上进行微调。本研究首次在斯瓦希里语和乌尔都语中开展此类分析，同时还在VL分析中引入了“推理依据”，其在评估中发挥了关键作用。
英文摘要：
Large language models (LLMs) have increased interest in vision language models (VLMs), which process image-text pairs as input. Studies investigating the visual understanding ability of VLMs have been proposed, but such studies are still preliminary because existing datasets do not permit a comprehensive evaluation of the fine-grained visual linguistic abilities of VLMs across multiple languages. To further explore the strengths of VLMs, such as GPT-4V \cite{openai2023GPT4}, we developed new datasets for the systematic and qualitative analysis of VLMs. Our contribution is four-fold: 1) we introduced nine vision-and-language (VL) tasks (including object recognition, image-text matching, and more) and constructed multilingual visual-text datasets in four languages: English, Japanese, Swahili, and Urdu through utilizing templates containing \textit{questions} and prompting GPT4-V to generate the \textit{answers} and the \textit{rationales}, 2) introduced a new VL task named \textit{unrelatedness}, 3) introduced rationales to enable human understanding of the VLM reasoning process, and 4) employed human evaluation to measure the suitability of proposed datasets for VL tasks. We show that VLMs can be fine-tuned on our datasets. Our work is the first to conduct such analyses in Swahili and Urdu. Also, it introduces \textit{rationales} in VL analysis, which played a vital role in the evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
