arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.03791cs.CL
收藏

M5——评估大型多模态模型在跨多语言与多文化视觉语言任务中性能的多样化基准

M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks

Universität Hamburg(汉堡大学);Microsoft Research India(微软研究院印度分院)

Florian Schneider, Sunayana Sitaram

AI总结:

本文提出M5,首个在多语言和多文化语境下评估大型多模态模型视觉语言任务的综合性基准,包含8个数据集、5项任务和41种语言,揭示了高低资源语言间的显著性能差异及模型规模并非决定性优势。

摘要:

自ChatGPT发布以来,自然语言处理领域经历了快速发展,尤其是大型语言模型(LLMs)及其多模态对应物大型多模态模型(LMMs)。尽管具备令人印象深刻的能力,但正如各种纯文本基准所证明的那样,LLMs在不同语言和文化语境中通常表现出显著的性能差异。然而,当前研究缺乏针对多模态视觉语言设置的此类基准。这项工作通过引入M5填补了这一空白,M5是首个旨在多语言和多文化语境下评估LMMs多样化视觉语言任务的综合性基准。M5包含覆盖5项任务和41种语言的8个数据集,重点关注代表性不足的语言和文化上多样化的图像。此外,我们引入了两个新数据集M5-VGR和M5-VLOD,包括一项新的视觉语言异常值检测任务,在该任务中所有被评估的开源模型都未能显著超越随机基线。通过广泛的评估和分析,我们强调了高资源语言与低资源语言之间存在的显著的任务无关性能差异。此外,我们展示了在多语言环境下,较大模型并不一定优于较小模型。

英文摘要:

Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only benchmarks. However, current research lacks such benchmarks for multimodal visio-linguistic settings. This work fills this gap by introducing M5, the first comprehensive benchmark designed to evaluate LMMs on diverse vision-language tasks within a multilingual and multicultural context. M5 includes eight datasets covering five tasks and $41$ languages, with a focus on underrepresented languages and culturally diverse images. Furthermore, we introduce two novel datasets, M5-VGR and M5-VLOD, including a new Visio-Linguistic Outlier Detection task, in which all evaluated open-source models fail to significantly surpass the random baseline. Through extensive evaluation and analyses, we highlight substantial task-agnostic performance disparities between high- and low-resource languages. Moreover, we show that larger models do not necessarily outperform smaller ones in a multilingual setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑