探索GPT-4V中的视觉文化意识：一项全面的探测
Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing
AI总结：
该研究使用MaRVL基准数据集，通过三项视觉任务全面探测了GPT-4V的视觉文化意识，发现其虽在低资源语言上表现较弱，但在图像描述中比原人工标注更具文化相关性。
摘要：
近年来，预训练的大型视觉语言模型因其卓越的性能而引起了广泛关注。尽管人们在从多个角度评估这些模型方面付出了大量努力，但最先进的GPT-4V模型在视觉文化意识方面的程度仍未被探索。为了填补这一空白，我们使用MaRVL基准数据集对GPT-4V进行了广泛探测，旨在调查其在以文化为侧重点的视觉理解方面的能力和局限性。具体而言，我们引入了三项视觉相关任务，即标题分类、成对标题生成和文化标签选择，以系统地深入进行细粒度的视觉文化评估。实验结果表明，GPT-4V在识别文化概念方面表现出色，但在低资源语言（如Tamil和Swahili）方面仍表现出较弱的性能。值得注意的是，通过人工评估，GPT-4V在图像描述任务中证明比原始的MaRVL人工标注更具文化相关性，这为未来的视觉文化基准构建提供了一种有前景的解决方案。
英文摘要：
Pretrained large Vision-Language models have drawn considerable interest in recent years due to their remarkable performance. Despite considerable efforts to assess these models from diverse perspectives, the extent of visual cultural awareness in the state-of-the-art GPT-4V model remains unexplored. To tackle this gap, we extensively probed GPT-4V using the MaRVL benchmark dataset, aiming to investigate its capabilities and limitations in visual understanding with a focus on cultural aspects. Specifically, we introduced three visual related tasks, i.e. caption classification, pairwise captioning, and culture tag selection, to systematically delve into fine-grained visual cultural evaluation. Experimental results indicate that GPT-4V excels at identifying cultural concepts but still exhibits weaker performance in low-resource languages, such as Tamil and Swahili. Notably, through human evaluation, GPT-4V proves to be more culturally relevant in image captioning tasks than the original MaRVL human annotations, suggesting a promising solution for future visual cultural benchmark construction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
