看见文化：视觉推理与定位的基准
Seeing Culture: A Benchmark for Visual Reasoning and Grounding
AI总结：
本文提出Seeing Culture Benchmark（SCB），通过两阶段视觉问答与分割任务，评估VLMs在东南亚文化图像上的推理与定位能力，揭示跨模态文化推理的不足。
摘要：
多模态视觉-语言模型（VLMs）在需要结合视觉与文本内容理解的各种任务中取得了显著进展，尤其是在文化理解任务中，随着新文化数据集的出现。然而，这些数据集常常缺乏文化推理能力，同时对许多文化的代表性不足。在本文中，我们引入了看见文化基准（SCB），专注于文化推理，采用一种新颖的方法，要求VLMs在两个阶段中对文化丰富的图像进行推理：i）通过多项选择视觉问答（VQA）选择正确的视觉选项，以及ii）分割相关的文化器物作为推理证据。第一阶段的视觉选项被系统地组织为三种类型：来自同一国家的选项、来自不同国家的选项，或混合组。值得注意的是，所有选项都源自每种类型的单一类别。只有在选择了正确的视觉选项后，才会进入第二阶段。SCB基准包含1,065张图像，捕捉了来自七个东南亚国家的138种文化器物，涵盖五个类别，这些国家的多元文化常常被忽视，并配有3,178个问题，其中1,093个是独特的，由人类注释者精心策划。我们对各种VLMs的评估揭示了跨模态文化推理的复杂性，并突显了在文化细微差别场景中视觉推理与空间定位之间的差距。SCB作为识别这些不足的关键基准，从而指导文化推理领域的未来发展。
英文摘要：
Multimodal vision-language models (VLMs) have made substantial progress in various tasks that require a combined understanding of visual and textual content, particularly in cultural understanding tasks, with the emergence of new cultural datasets. However, these datasets frequently fall short of providing cultural reasoning while underrepresenting many cultures. In this paper, we introduce the Seeing Culture Benchmark (SCB), focusing on cultural reasoning with a novel approach that requires VLMs to reason on culturally rich images in two stages: i) selecting the correct visual option with multiple-choice visual question answering (VQA), and ii) segmenting the relevant cultural artifact as evidence of reasoning. Visual options in the first stage are systematically organized into three types: those originating from the same country, those from different countries, or a mixed group. Notably, all options are derived from a singular category for each type. Progression to the second stage occurs only after a correct visual option is chosen. The SCB benchmark comprises 1,065 images that capture 138 cultural artifacts across five categories from seven Southeast Asia countries, whose diverse cultures are often overlooked, accompanied by 3,178 questions, of which 1,093 are unique and meticulously curated by human annotators. Our evaluation of various VLMs reveals the complexities involved in cross-modal cultural reasoning and highlights the disparity between visual reasoning and spatial grounding in culturally nuanced scenarios. The SCB serves as a crucial benchmark for identifying these shortcomings, thereby guiding future developments in the field of cultural reasoning. https://github.com/buraksatar/SeeingCulture
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
