CultureVLM：面向100多个国家的视觉-语言模型文化理解能力的刻画与提升
CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
AI总结：
本文构建了覆盖188个国家/地区的大规模多元文化基准CultureVerse，并提出在其上微调的CultureVLM系列模型，显著提升了视觉-语言模型的文化理解能力，同时保持通用性能不下降。
摘要：
视觉-语言模型（VLMs）推动了人机交互的发展，但在文化理解方面仍存在困难，由于训练数据以西方为中心而存在偏差，常常误读符号、手势和文化器物。本文构建了CultureVerse，一个大规模多模态基准，涵盖19,682个文化概念、188个国家/地区、15个文化概念类别和3种问题类型，旨在刻画和提升视觉-语言模型的多元文化理解能力。随后，我们提出CultureVLM，一系列在我们的数据集上微调的视觉-语言模型，在文化理解方面取得了显著的性能提升。我们对16个模型的评估揭示了显著差异：在西方概念上表现更强，而在非洲和亚洲语境中表现较弱。在CultureVerse上的微调增强了文化感知能力，展现出跨文化、跨大洲和跨数据集的泛化能力，同时没有牺牲模型在通用视觉-语言模型基准上的性能。我们进一步提出了关于文化泛化与遗忘的见解。我们希望这项工作能够为构建更加公平且具有文化意识的多元模态人工智能系统奠定基础。
英文摘要：
Vision-language models (VLMs) have advanced human-AI interaction but struggle with cultural understanding, often misinterpreting symbols, gestures, and artifacts due to biases in predominantly Western-centric training data. In this paper, we construct CultureVerse, a large-scale multimodal benchmark covering 19, 682 cultural concepts, 188 countries/regions, 15 cultural concepts, and 3 question types, with the aim of characterizing and improving VLMs' multicultural understanding capabilities. Then, we propose CultureVLM, a series of VLMs fine-tuned on our dataset to achieve significant performance improvement in cultural understanding. Our evaluation of 16 models reveals significant disparities, with a stronger performance in Western concepts and weaker results in African and Asian contexts. Fine-tuning on our CultureVerse enhances cultural perception, demonstrating cross-cultural, cross-continent, and cross-dataset generalization without sacrificing performance on models' general VLM benchmarks. We further present insights on cultural generalization and forgetting. We hope that this work could lay the foundation for more equitable and culturally aware multimodal AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
