arXiv 2510.052912026-09-02cs.CL

Camellia: 亚洲语言中LLMs文化偏见的基准测试

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

Georgia Institute of Technology（佐治亚理工学院）；Samsung R&D Institute Philippines（三星菲律宾研发院）；Sungkyunkwan University（成均馆大学）；Tohoku University（东北大学）；National University of Singapore（新加坡国立大学）；University of Copenhagen（哥本哈根大学）；University of Michigan（密歇根大学）；Indian Institute of Technology Delhi（印度理工学院德里）；Institute of Science Tokyo（东京科学研究院）

Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, Neel Kothari, Sahajpreet Singh, Sarah M… 展开作者

Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, Neel Kothari, Sahajpreet Singh, Sarah Masud, Tanish Patwa, Trung Thanh Tran, Zohaib Khan, Alan Ritter, Tanmoy Chakraborty, Yuki Arase, Keisuke Sakaguchi, JinYeong Bak, Wei Xu

备注：: Accepted at EMNLP 2026 Main Conference

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出Camellia基准，通过三个任务评估九种亚洲语言中多语言大模型对亚洲与西方文化实体的偏见，发现模型存在文化适应困难、情感关联差异及实体提取性能差距。

摘要：

随着大语言模型（LLMs）多语言能力的增强，它们对文化多样性实体的敏感性变得越来越重要。Naous等人（2024）的前期工作表明，LLMs在阿拉伯语中往往偏好与西方相关的实体。由于缺乏以实体为中心的多语言基准，这种偏见是否也存在于各种非西方语言中尚不清楚。在本文中，我们介绍了Camellia，这是一个用于评估九种亚洲语言（涵盖六种亚洲文化）中实体中心文化偏见的基准。Camellia包括19,530个手动注释的实体，这些实体与所涵盖的亚洲或西方文化相关，以及从社交媒体帖子中提取的2,173个这些实体的掩码上下文。利用Camellia，我们在三个任务中评估了四个最近的多语言LLMs的文化偏见：文化上下文适应、情感关联和实体抽取式问答。我们的分析表明，LLMs在这些语言中难以进行文化适应，不同地区开发的模型表现存在差异。我们进一步观察到，不同的LLM家族可能持有不同的偏见，这反映在它们将文化与特定情感联系起来的方式上。最后，我们发现LLMs在某些亚洲语言中可能难以理解上下文，从而在实体抽取中造成文化之间的性能差距。

英文摘要：

As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western languages. In this paper, we introduce Camellia, a benchmark for evaluating entity-centric cultural biases in nine Asian languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian languages, creating performance gaps between cultures in entity extraction.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。