arXiv 2502.128522025-02-18cs.CL

MVL-SIB：用于跨模态主题匹配的大规模多语言视觉-语言基准

MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

Center for Artificial Intelligence and Data Science, University of Würzburg（维尔茨堡大学人工智能与数据科学中心）；Language Technology Group, University of Hamburg（汉堡大学语言技术组）

Fabian David Schmidt, Florian Schneider, Chris Biemann, Goran Glavaš

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有多语言视觉-语言基准覆盖语言少、低资源语言评估缺失的问题，本文提出覆盖205种语言的MVL-SIB基准，测试多种模型后揭示低资源语言跨模态匹配性能差等问题，为模型评估提供综合探针。

摘要：

现有多语言视觉-语言（VL）基准通常仅覆盖少数语言。因此，大型视觉-语言模型（LVLMs）的评估主要针对高资源语言，凸显了对低资源语言评估数据的需求。为解决这一局限，我们引入MVL-SIB，一个大规模多语言视觉-语言基准，可评估205种语言的跨模态和纯文本主题匹配——比现有最多语言的VL基准多100余种。我们随后在MVL-SIB上对一系列开源权重LVLMs及GPT-4o（-mini）进行基准测试。结果显示，LVLMs在低资源语言的跨模态主题匹配中表现不佳，如在N'Koo语言上的表现与随机水平相当。我们的分析进一步表明，对于低资源语言，LVLMs的VL支持相对于文本支持的下降程度不成比例，这可通过跨模态与纯文本主题匹配性能的比较得到证明。我们还观察到，开源权重LVLMs无法从用多张图像表示主题中获益，表明这些模型在处理多图像任务方面尚未完全有效。通过将MVL-SIB上的性能与其他多语言VL基准相关联，我们强调MVL-SIB可作为LVLMs多语言VL理解的综合探针。

英文摘要：

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for low-resource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages -- over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multi-image tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。