arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.02067cs.CL
收藏

大陆交汇处:利用大型多模态模型进行面向文化适应的自动化文化元素提取

Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models

George Mason University(乔治梅森大学)

Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos

AI总结:

本文通过构建 DalleStreet 数据集并提取超 1.8 万个文化元素,系统评估了 LMMs 的文化理解差异,并提出 CultureAdapt 流水线以自动适配图像中的文化表征。

摘要:

我们提出了一项全面的三阶段研究,旨在考察:(1) 大型多模态模型(LMMs)的文化理解能力,为此我们引入了 DalleStreet——一个由 DALL-E 3 生成并经人工验证的大规模数据集,包含 67 个国家和 10 个概念类别的 9,935 张图像;(2) 通过文化元素提取任务揭示的潜在隐性且可能带有刻板印象的文化关联;(3) 一种基于提取到的关联、利用模块化流水线 CultureAdapt 对图像中的文化表征进行适配的方法。我们在 DalleStreet 及其他现有基准上发现,开源模型(LLaVA)和闭源模型(GPT-4V)在地理次区域层面的文化理解上存在差异,并尝试利用我们识别出的与不同国家相关联的超过 18,000 个文化元素来理解这些差异。我们的研究结果揭示了 LMMs 文化能力的一幅细致图景,凸显了开发具备文化感知能力的系统的必要性。数据集和代码可在 https://github.com/iamshnoo/crossroads 获取。

英文摘要:

We present a comprehensive three-phase study to examine (1) the cultural understanding of Large Multimodal Models (LMMs) by introducing DalleStreet, a large-scale dataset generated by DALL-E 3 and validated by humans, containing 9,935 images of 67 countries and 10 concept classes; (2) the underlying implicit and potentially stereotypical cultural associations with a cultural artifact extraction task; and (3) an approach to adapt cultural representation in an image based on extracted associations using a modular pipeline, CultureAdapt. We find disparities in cultural understanding at geographic sub-region levels with both open-source (LLaVA) and closed-source (GPT-4V) models on DalleStreet and other existing benchmarks, which we try to understand using over 18,000 artifacts that we identify in association to different countries. Our findings reveal a nuanced picture of the cultural competence of LMMs, highlighting the need to develop culture-aware systems. Dataset and code are available at https://github.com/iamshnoo/crossroads

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑