面向文化理解的视觉语言模型基准测试
Benchmarking Vision Language Models for Cultural Understanding
AI总结：
该研究提出CulturalVQA基准，包含跨5大洲11个国家的2378个图像-问题对，用于评估视觉语言模型（VLMs）的多元文化理解能力，测试发现GPT-4V和Gemini等模型在北美文化理解表现强而非洲弱，且不同文化维度表现存在差异。
摘要：
基础模型和视觉语言预训练显著推进了视觉语言模型（VLMs），使其能够对视觉和语言数据进行多模态处理。然而，它们的性能通常是在一般场景理解（即识别物体、属性和动作）上进行评估，而非文化理解。本研究引入了CulturalVQA，这是一个视觉问答基准，旨在评估VLM的地理多样性文化理解能力。我们精心整理了包含2,378个图像-问题对的数据集，每个问题配有1-5个答案，代表了跨越5大洲11个国家的文化。这些问题探究了对文化各个方面的理解，如服装、食物、饮料、仪式和传统。在CulturalVQA上对VLM（包括GPT-4V和Gemini）进行基准测试，揭示了它们在不同地区文化理解水平上的差异：对北美的文化理解能力很强，而对非洲的性能则显著较低。我们还观察到它们在不同文化方面的性能也存在差异，服装、仪式和传统的表现高于食物和饮料。这些差异帮助我们识别了VLM在文化理解方面的不足，并证明了CulturalVQA作为评估VLM在理解多元文化方面进展的综合评估集的潜力。
英文摘要：
Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of visual and linguistic data. However, their performance has been typically assessed on general scene understanding - recognizing objects, attributes, and actions - rather than cultural comprehension. This study introduces CulturalVQA, a visual question-answering benchmark aimed at assessing VLM's geo-diverse cultural understanding. We curate a collection of 2,378 image-question pairs with 1-5 answers per question representing cultures from 11 countries across 5 continents. The questions probe understanding of various facets of culture such as clothing, food, drinks, rituals, and traditions. Benchmarking VLMs on CulturalVQA, including GPT-4V and Gemini, reveals disparity in their level of cultural understanding across regions, with strong cultural understanding capabilities for North America while significantly lower performance for Africa. We observe disparity in their performance across cultural facets too, with clothing, rituals, and traditions seeing higher performances than food and drink. These disparities help us identify areas where VLMs lack cultural understanding and demonstrate the potential of CulturalVQA as a comprehensive evaluation set for gauging VLM progress in understanding diverse cultures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
