arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.10001cs.CLcs.AIcs.LG
收藏

利用Wikidata实现地理导向的社会文化偏见数据集创建:应用于拉丁美洲

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

Yannis Karmim, Renato Pino, Hernan Contreras, Hernan Lira, Sebastian Cifuentes, Simon Escoffier, Luis Martí, Djamé Seddah, Valentin Barrière

AI总结:

本文利用Wikidata和维基百科内容创建拉丁美洲社会文化偏见数据集,通过多项选择题评估LLMs在不同文化背景下的表现差异,发现模型在原生语言中表现更佳,且伊比利亚西班牙文化认知度高于拉丁美洲文化。

摘要:

大型语言模型(LLMs)在各种文化背景下表现出不平等。大多数著名的开放权重模型是在全球北半球数据上训练的,并对其他文化表现出偏见行为。此外,检测非英语语言中的偏见的资源明显不足,尤其是拉丁美洲(Latam),该大陆包含多种文化,尽管它们共享一个共同的文化基础。我们提出利用维基百科的内容、Wikidata知识图谱的结构以及社会科学专家的知识,以创建基于不同流行和社会文化的各种拉丁美洲国家的问题/答案(Q/As)对数据集。我们创建了包含超过26,000个问题及其相关答案的数据集,这些答案是从26,000篇维基百科文章中提取并转换为西班牙语和葡萄牙语的多项选择题(MCQ),然后翻译成英语。我们使用这个MCQ来量化各种LLMs的知识程度,并发现(i)拉丁美洲国家之间表现存在差异,一些国家比其他国家更容易被大多数模型处理;(ii)模型在原生语言中表现更好;(iii)伊比利亚西班牙文化比拉丁美洲文化更受认可。

英文摘要:

Large Language Models (LLMs) exhibit inequalities with respect to various cultural contexts. Most prominent open-weights models are trained on Global North data and show prejudicial behavior towards other cultures. Moreover, there is a notable lack of resources to detect biases in non-English languages, especially from Latin America (Latam), a continent containing various cultures, even though they share a common cultural ground. We propose to leverage the content of Wikipedia, the structure of the Wikidata knowledge graph, and expert knowledge from social science in order to create a dataset of question/answer (Q/As) pairs, based on the different popular and social cultures of various Latin American countries. We create the LatamQA database of over 26k questions and associated answers extracted from 26k Wikipedia articles, and transformed into multiple-choice questions (MCQ) in Spanish and Portuguese, in turn translated to English. We use this MCQ to quantify the degree of knowledge of various LLMs and find out (i) a discrepancy in performances between the Latam countries, ones being easier than others for the majority of the models, (ii) that the models perform better in their original language, and (iii) that Iberian Spanish culture is better known than Latam one.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑