VLURes：低资源语言下VLM视觉与语言理解能力的基准测试
VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
AI总结：
VLURes是一个多语言基准，包含八个视觉语言任务和一个无关性任务，评估VLM在英语、日语及低资源语言斯瓦希里语和乌尔都语中的细粒度理解能力；最佳模型GPT-4o准确率90.8%，落后人类6.7%，凸显了智能体多模态推理的挑战。
摘要：
视觉语言模型（VLM）对于提升智能体的感知能力至关重要。然而，VLM的评估目前仍主要局限于以英语为中心的基准测试，其中图像-文本对多为短文本。为了在长文本设置下评估VLM在四种语言中的细粒度能力，我们引入了一个新颖的多语言基准VLURes，包含八个视觉与语言任务，以及一个开创性的无关性任务，用以探测VLM在英语、日语以及低资源语言斯瓦希里语和乌尔都语中的细粒度视觉与语言理解能力。我们的数据集从目标语言的网络资源中整理而成，涵盖十个不同的图像类别和丰富的文本上下文，为斯瓦希里语和乌尔都语引入了宝贵的视觉语言资源。通过提示VLM生成响应和理由，并由自动评估和母语者评估，我们揭示了在智能体至关重要的任务（如物体识别、场景理解和关系理解）上，不同语言和任务间的性能差异。我们使用VLURes对十个VLM进行了评估。表现最佳的模型GPT-4o达到了90.8%的总体准确率，落后人类表现6.7%，而开源模型的差距更大。这一差距凸显了VLURes在开发智能体以应对多模态视觉推理方面的关键作用。
英文摘要：
Vision Language Models (VLMs) are pivotal for advancing perception in intelligent agents. Yet, evaluation of VLMs remains limited to predominantly English-centric benchmarks in which the image-text pairs comprise short texts. To evaluate VLM fine-grained abilities, in four languages under long-text settings, we introduce a novel multilingual benchmark VLURes featuring eight vision-and-language tasks, and a pioneering unrelatedness task, to probe the fine-grained Visual and Linguistic Understanding capabilities of VLMs across English, Japanese, and low-resource languages, Swahili, and Urdu. Our datasets, curated from web resources in the target language, encompass ten diverse image categories and rich textual context, introducing valuable vision-language resources for Swahili and Urdu. By prompting VLMs to generate responses and rationales, evaluated automatically and by native speakers, we uncover performance disparities across languages and tasks critical to intelligent agents, such as object recognition, scene understanding, and relationship understanding. We conducted evaluations of ten VLMs with VLURes. The best performing model, GPT-4o, achieves an overall accuracy of 90.8% and lags human performance by 6.7%, though the gap is larger for open-source models. The gap highlights VLURes' critical role in developing intelligent agents to tackle multi-modal visual reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
