arXiv 2601.140632026-06-06cs.CLcs.AIcs.CY

XCR-Bench：通过文化特定项目和霍尔三元组对大型语言模型进行跨文化推理基准测试

XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad

Department of Computer Science, National Centre for Text Mining, The University of Manchester（计算机科学系，国家文本挖掘中心，曼彻斯特大学）；ELLIS Manchester（曼彻斯特ELLIS）；School of Computing, Queen’s University, Ontario, Canada（计算学院，加拿大皇后大学）；Computer Science, University of Illinois Chicago（计算机科学，伊利诺伊大学芝加哥分校）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）；Department of Computer Science and Artificial Intelligence, Umm Al-Qura University, Makkah, Saudi Arabia（计算机科学与人工智能系，乌姆·阿勒·卡拉大学，麦加，沙特阿拉伯）

Mohsinul Kabir, Tasnim Ahmed, Md Mezbaur Rahman, Shaoxiong Ji, Hassan Alhuzali, Yuechen Jiang, Jimin Huang, Sophia Ananiadou

备注：: Under Review

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出XCR-Bench基准，包含4.1k平行句和1098个文化特定项目，结合Newmark框架与霍尔三元组评估LLM跨文化推理，发现模型在深层文化层面表现显著下降，且存在区域和民族宗教偏见。

摘要：

大型语言模型（LLM）的跨文化能力需要理解并适应不同文化背景下的文化特定项目（CSI）。然而，由于缺乏高质量且带有平行跨文化句子对的CSI标注语料库，评估该能力的进展仍然有限。我们引入了XCR-Bench，一个跨文化推理基准，包含4.1k个平行句子和1,098个CSI，涵盖三个推理任务。XCR-Bench将Newmark的CSI框架与霍尔文化三元组相结合，从而能够评估从可观察实践到隐性社会规范和价值观等不同文化可见性层面的能力。对八个多语言LLM的实验表明，最先进的模型在识别和适应特定CSI类别方面表现出持续的弱点，揭示了表面召回与显式文化推理之间的差距。在文化敏感类别和更深文化层面上，性能显著下降（p<0.005，8/8模型），并且适应质量在不同目标文化和孟加拉语区域变体之间系统性变化，表明即使在单一语言环境中也存在编码的区域和民族宗教偏见。我们公开发布语料库和代码，以支持未来跨文化NLP的研究。

英文摘要：

Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts. However, progress in evaluating this capability remains limited by the lack of high-quality CSI-annotated corpora with parallel cross-cultural sentence pairs. We introduce XCR-Bench, a Cross(X)-Cultural Reasoning Benchmark containing 4.1k parallel sentences and 1,098 CSIs across three reasoning tasks. XCR-Bench integrates Newmark's CSI framework with Hall's Triad of Culture, enabling evaluation across levels of cultural visibility -- from observable practices to implicit social norms and values. Experiments on eight multilingual LLMs show that state-of-the-art models exhibit consistent weaknesses in identifying and adapting specific categories of CSIs, revealing a gap between surface-level recall and explicit cultural reasoning. Performance declines significantly on culturally sensitive categories and deeper cultural levels (p<0.005, 8/8 models), and adaptation quality varies systematically across target cultures and Bengali regional variants, indicating encoded regional and ethno-religious biases even within a single linguistic setting. We publicly release the corpus and code to support future research on cross-cultural NLP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。