大型语言模型在多大程度上能够识别多样性中的文化统一性？
How Well Do LLMs Identify Cultural Unity in Diversity?
AI总结：
本文构建包含10国285个文化概念、1,425个样例的 CUNIT 基准，通过对比匹配任务评估 LLM 识别跨文化概念共性的能力，发现其表现仍明显弱于人类且受地域邻近性影响较弱。
摘要：
关于大型语言模型（LLM）文化意识的许多研究都聚焦于模型对地域文化多样性的敏感性。然而，除了跨文化差异之外，不同文化之间也存在共同基础。例如，美国的新娘头纱与中国的红盖头在文化相关功能上相似。在本研究中，我们引入了一个基准数据集 CUNIT，用于评估 decoder-only LLM 对概念文化统一性的理解。具体而言，CUNIT 包含 1,425 个评估样例，这些样例建立在来自 10 个国家的 285 个传统文化特有概念之上。基于对每个概念的文化相关特征进行系统化人工标注，我们计算任意一对跨文化概念之间的文化关联。依托该数据集，我们设计了一个对比匹配任务，以评估 LLM 识别高度相关跨文化概念对的能力。我们在 CUNIT 上评估了 3 个强大的 LLM，使用 3 种流行的提示策略，并分别设置提供所有抽取的概念特征或完全不提供特征两种条件。有趣的是，我们发现各国在服饰概念上的文化关联与食物概念存在显著差异。我们的分析表明，与人类相比，LLM 在捕捉概念之间的跨文化关联方面仍然有限。此外，地域文化邻近性对模型捕捉跨文化关联的性能影响较弱。
英文摘要：
Much work on the cultural awareness of large language models (LLMs) focuses on the models' sensitivity to geo-cultural diversity. However, in addition to cross-cultural differences, there also exists common ground across cultures. For instance, a bridal veil in the United States plays a similar cultural-relevant role as a honggaitou in China. In this study, we introduce a benchmark dataset CUNIT for evaluating decoder-only LLMs in understanding the cultural unity of concepts. Specifically, CUNIT consists of 1,425 evaluation examples building upon 285 traditional cultural-specific concepts across 10 countries. Based on a systematic manual annotation of cultural-relevant features per concept, we calculate the cultural association between any pair of cross-cultural concepts. Built upon this dataset, we design a contrastive matching task to evaluate the LLMs' capability to identify highly associated cross-cultural concept pairs. We evaluate 3 strong LLMs, using 3 popular prompting strategies, under the settings of either giving all extracted concept features or no features at all on CUNIT Interestingly, we find that cultural associations across countries regarding clothing concepts largely differ from food. Our analysis shows that LLMs are still limited to capturing cross-cultural associations between concepts compared to humans. Moreover, geo-cultural proximity shows a weak influence on model performance in capturing cross-cultural associations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
