GIMMICK——全球包容性多模态多任务文化知识基准评测
GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking
AI总结：
本文提出覆盖144国、六项任务和728个文化侧面的 GIMMICK 多模态基准，评测31个模型，揭示 LVLM/LLM 的西方文化偏见及规模、模态与地理线索的影响。
摘要：
大型视觉-语言模型（LVLM）近来因其出色性能和广泛适用性而受到关注。尽管此前已有研究表明，它们在涉及非西方语境的使用场景中效果不足，但现有研究的范围仍然有限：仅覆盖狭窄的文化范围，只关注少数文化方面，或仅在单一任务上评估有限的若干模型。为推动具有全球包容性的 LVLM 研究，我们提出 GIMMICK，这是一个大规模多模态基准，旨在评估代表全球六个宏观区域的 144 个国家的广泛文化知识。GIMMICK 包含基于三个新数据集构建的六项任务，这些数据集涵盖 728 个独特的文化事件或文化侧面；我们在其上评估了 20 个 LVLM 和 11 个 LLM，其中包括 5 个专有模型和 26 个各种规模的开放权重模型。我们系统考察了：（1）区域文化偏见，（2）模型规模的影响，（3）输入模态，以及（4）外部线索。分析表明，各模型和任务均存在强烈偏向西方文化的倾向，并凸显模型规模与性能之间的强相关性，以及多模态输入和外部地理线索的有效性。我们还发现，模型对有形文化方面的知识多于无形方面（例如食物与仪式相比），并且它们擅长识别宽泛的文化起源，却在更细致的理解上表现困难。
英文摘要：
Large Vision-Language Models (LVLMs) have recently gained attention due to their distinctive performance and broad applicability. While it has been previously shown that their efficacy in usage scenarios involving non-Western contexts falls short, existing studies are limited in scope, covering just a narrow range of cultures, focusing exclusively on a small number of cultural aspects, or evaluating a limited selection of models on a single task only. Towards globally inclusive LVLM research, we introduce GIMMICK, an extensive multimodal benchmark designed to assess a broad spectrum of cultural knowledge across 144 countries representing six global macro-regions. GIMMICK comprises six tasks built upon three new datasets that span 728 unique cultural events or facets on which we evaluated 20 LVLMs and 11 LLMs, including five proprietary and 26 open-weight models of all sizes. We systematically examine (1) regional cultural biases, (2) the influence of model size, (3) input modalities, and (4) external cues. Our analyses reveal strong biases toward Western cultures across models and tasks and highlight strong correlations between model size and performance, as well as the effectiveness of multimodal input and external geographic cues. We further find that models have more knowledge of tangible than intangible aspects (e.g., food vs. rituals) and that they excel in recognizing broad cultural origins but struggle with a more nuanced understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
