MAKIEval：一个基于Wikidata的多语言自动文化意识评估框架，用于大语言模型
MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs
AI总结：
本文提出基于Wikidata的多语言自动评估框架MAKIEval，通过四个指标评估LLMs在13种语言、19个国家和地区及6个文化主题上的文化意识，发现模型在英语中表现更强。
摘要：
大语言模型（LLMs）在全球范围内被用于多种语言，但其以英语为中心的预训练引发了人们对跨语言文化意识差异的担忧，这往往导致输出存在偏见。然而，由于基准有限且翻译质量存疑，全面的多语言评估仍然具有挑战性。为了更好地评估这些差异，我们提出了MAKIEval，这是一个自动化的多语言框架，用于评估LLMs在不同语言、地区和主题上的文化意识。MAKIEval评估开放式文本生成，捕捉模型如何用自然语言表达具有文化根基的知识。利用Wikidata的多语言结构作为跨语言锚点，它自动识别模型输出中的文化实体，并将其链接到结构化知识，从而无需人工标注或翻译即可实现可扩展的、语言无关的评估。随后，我们引入了四个指标，用于捕捉文化意识的互补维度：粒度、多样性、文化特异性以及跨语言一致性。我们评估了来自世界不同地区的7个LLMs，涵盖开源和专有系统，涉及13种语言、19个国家和地区以及6个具有文化显著性的主题（例如食物、服饰）。值得注意的是，我们发现模型往往在英语中表现出更强的文化意识，这表明英语提示能更有效地激活具有文化根基的知识。
英文摘要：
Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness, often resulting in biased outputs. However, comprehensive multilingual evaluation remains challenging due to limited benchmarks and questionable translation quality. To better assess these disparities, we introduce MAKIEval, an automatic multilingual framework for evaluating cultural awareness in LLMs across languages, regions, and topics. MAKIEval evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. Leveraging Wikidata's multilingual structure as a cross-lingual anchor, it automatically identifies cultural entities in model outputs and links them to structured knowledge, enabling scalable, language-agnostic evaluation without manual annotation or translation. We then introduce four metrics that capture complementary dimensions of cultural awareness: granularity, diversity, cultural specificity, and consensus across languages. We assess 7 LLMs developed from different parts of the world, encompassing both open-source and proprietary systems, across 13 languages, 19 countries and regions, and 6 culturally salient topics (e.g., food, clothing). Notably, we find that models tend to exhibit stronger cultural awareness in English, suggesting that English prompts more effectively activate culturally grounded knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
