MCEval：一种用于 LLM 公平多语言文化评估的动态框架
MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs
AI总结：
提出多语言文化评估框架 MCEval，通过动态文化问题构建和因果分析评估 13 种文化与语言下的 LLM 文化意识与偏见，揭示了模型性能与训练数据、语言-文化对齐的关系及跨语言公平性问题。
摘要：
大型语言模型表现出文化偏见和有限的跨文化理解能力，特别是在服务多样化的全球用户群体时。我们提出 MCEval，一种新颖的多语言评估框架，它采用动态文化问题构建，并通过反事实重述和混淆因子重述实现因果分析。我们的综合评估涵盖 13 种文化和 13 种语言，系统地评估了不同语言场景下的文化意识和文化偏见。该框架提供 39,897 个文化意识实例和 17,940 个文化偏见实例。实验结果揭示了不同语言场景下的性能差异，表明最佳文化表现不仅与训练数据分布有关，还与语言-文化对齐相关。评估结果还暴露了公平性问题，在英语场景中看似成功的方法造成了显著劣势。MCEval 是首个全面的多语言文化评估框架，为 LLM 的文化理解提供了更深刻的见解。
英文摘要：
Large language models exhibit cultural biases and limited cross-cultural understanding capabilities, particularly when serving diverse global user populations. We propose MCEval, a novel multilingual evaluation framework that employs dynamic cultural question construction and enables causal analysis through Counterfactual Rephrasing and Confounder Rephrasing. Our comprehensive evaluation spans 13 cultures and 13 languages, systematically assessing both cultural awareness and cultural bias across different linguistic scenarios. The framework provides 39,897 cultural awareness instances and 17,940 cultural bias instances. Experimental results reveal performance disparities across different linguistic scenarios, demonstrating that optimal cultural performance is not only linked to training data distribution, but also is related to language-culture alignment. The evaluation results also expose the fairness issue, where approaches appearing successful in the English scenario create substantial disadvantages. MCEval represents the first comprehensive multilingual cultural evaluation framework that provides deeper insights into LLMs' cultural understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
