Language Models (LMs) have been shown to exhibit a strong preference towards entities associated with Western culture when operating in non-Western languages. In this paper, we aim to uncover the origins of entity-related cultural biases in LMs by analyzing several contributing factors, including the representation of entities in pre-training data and the impact of variations in linguistic phenomena across languages. We introduce CAMeL-2, a parallel Arabic-English benchmark of 58,086 entities associated with Arab and Western cultures and 367 masked natural contexts for entities. Our evaluations using CAMeL-2 reveal reduced performance gaps between cultures by LMs when tested in English compared to Arabic. We find that LMs struggle in Arabic with entities that appear at high frequencies in pre-training, where entities can hold multiple word senses. This also extends to entities that exhibit high lexical overlap with languages that are not Arabic but use the Arabic script. Further, we show how frequency-based tokenization leads to this issue in LMs, which gets worse with larger Arabic vocabularies. We will make CAMeL-2 available at: https://github.com/tareknaous/camel2
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
Cohere For AI；EPFL（洛桑联邦理工学院）；Hugging Face；Mila；McGill University（麦吉尔大学）；AI Singapore（新加坡人工智能研究院）；National University of Singapore（新加坡国立大学）；MIT（麻省理工学院）；KAIST（韩国科学技术院）；Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Carnegie Mellon University（卡内基梅隆大学）；CONICET & Universidad de Buenos Aires（阿根廷科学研究与技术委员会 & 布宜诺斯艾利斯大学）；Meta AI Research（Meta 人工智能研究院）
该研究分析翻译版 MMLU 的语言伪影与西方文化偏见，提出覆盖 42 种语言并经专业与社区标注核验的 Global MMLU，以支持更公平的多语言模型评估。
BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India（印度理工学院帕纳加邦计算机科学与工程系）；Indian Institute of Technology Patna, India（印度理工学院帕纳加邦）
It is tempting to treat frequency trends from the Google Books data sets as indicators of the "true" popularity of various words and phrases. Doing so allows us to draw quantitatively strong conclusions about the evolution of cultural perception of a given topic, such as time or gender. However, the Google Books corpus suffers from a number of limitations which make it an obscure mask of cultural popularity. A primary issue is that the corpus is in effect a library, containing one of each book. A single, prolific author is thereby able to noticeably insert new phrases into the Google Books lex…
While cultural backgrounds have been shown to affect linguistic expressions, existing natural language processing (NLP) research on culture modeling is overly coarse-grained and does not examine cultural differences among speakers of the same language. To address this problem and augment NLP models with cultural background features, we collect, annotate, manually validate, and benchmark EnCBP, a finer-grained news-based cultural background prediction dataset in English. Through language modeling (LM) evaluations and manual analyses, we confirm that there are noticeable differences in linguisti…
When building NLP models, there is a tendency to aim for broader coverage, often overlooking cultural and (socio)linguistic nuance. In this position paper, we make the case for care and attention to such nuances, particularly in dataset annotation, as well as the inclusion of cultural and linguistic expertise in the process. We present a playbook for responsible dataset creation for polyglossic, multidialectal languages. This work is informed by a study on Arabic annotation of social media content.
Digital hybridity and relics in cultural heritage: using corpus linguistics to inform design in emerging technologies from AI to VR
School of English, University of Nottingham（诺丁汉大学英语学院）；School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）；Department of Modern Languages, Aberystwyth University（阿伯里斯特威斯大学现代语言系）
Word choice is dependent on the cultural context of writers and their subjects. Different words are used to describe similar actions, objects, and features based on factors such as class, race, gender, geography and political affinity. Exploratory techniques based on locating and counting words may, therefore, lead to conclusions that reinforce culturally inflected boundaries. We offer a new method, the DualNeighbors algorithm, for linking thematically similar documents both within and across discursive and linguistic barriers to reveal cross-cultural connections. Qualitative and quantitative …
CENIDET（墨西哥国家数字教育技术中心）；COLMEX（墨西哥学院）；IIM（印度管理研究所）；Université de Montréal（蒙特利尔大学）；EBSI（图书馆与信息科学学院）；Université d’Avignon（阿维尼翁大学）；LIA（阿维尼翁计算机科学实验室）
This work aims to evaluate the ability that both probabilistic and state-of-the-art vector space modeling (VSM) methods provide to well known machine learning algorithms to identify social network documents to be classified as aggressive, gender biased or communally charged. To this end, an exploratory stage was performed first in order to find relevant settings to test, i.e. by using training and development samples, we trained multiple algorithms using multiple vector space modeling and probabilistic methods and discarded the less informative configurations. These systems were submitted to t…
Words shift in meaning for many reasons, including cultural factors like new technologies and regular linguistic processes like subjectification. Understanding the evolution of language and culture requires disentangling these underlying causes. Here we show how two different distributional measures can be used to detect two different types of semantic change. The first measure, which has been used in many previous works, analyzes global shifts in a word's distributional semantics, it is sensitive to changes due to regular processes of linguistic drift, such as the semantic generalization of p…
Carnegie Mellon University（卡内基梅隆大学）；Seoul National University（首尔国立大学）；Sogang University（西江大学）
Much work in cross-lingual transfer learning explored how to select better transfer languages for multilingual tasks, primarily focusing on typological and genealogical similarities between languages. We hypothesize that these measures of linguistic proximity are not enough when working with pragmatically-motivated tasks, such as sentiment analysis. As an alternative, we introduce three linguistic features that capture cross-cultural similarities that manifest in linguistic patterns and quantify distinct aspects of language pragmatics: language context-level, figurative language, and the lexif…
The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models
University College London（伦敦大学学院）；Queen’s University Belfast（贝尔法斯特女王大学）；The University of Tokyo（东京大学）；University of Bisha（比沙大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；AI Centre, University College London（伦敦大学学院人工智能中心）
HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMs
Kyushu University（九州大学）；hon9kon9ize；The Education University of Hong Kong（香港教育大学）；The University of Hong Kong（香港大学）；Independent Researcher（独立研究者）；Votee AI；Beever AI
Cultural Biases of Large Language Models and Humans in Historical Interpretation
Research & Development Department of Information Security and Communication Technology（信息安全和通信技术研发部）；Maggioli SpA（Maggioli股份公司）；Norwegian University of Science and Technology - NTNU（挪威科技大学）
MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西溪大学工程学院）；Southern University of Science and Technology（南方科技大学）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西溪研究院先进技术研究所）
MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；East China Normal University（东华大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Institute of High Performance Computing, A*STAR（高性能计算研究所，A*STAR）
Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Georgia Institute of Technology（佐治亚理工学院）；Samsung R&D Institute Philippines（三星菲律宾研发院）；Sungkyunkwan University（成均馆大学）；Tohoku University（东北大学）；National University of Singapore（新加坡国立大学）；University of Copenhagen（哥本哈根大学）；University of Michigan（密歇根大学）；Indian Institute of Technology Delhi（印度理工学院德里）；Institute of Science Tokyo（东京科学研究院）
From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher
Department of Computer Science, King Khalid University（国王 Khalid 大学计算机科学系）；Department of Computer Science and Artificial Intelligence, Umm Al-Qura University（乌姆·阿勒·库拉大学计算机科学与人工智能系）；Imam Abdulrahman Bin Faisal University（伊玛目阿卜杜勒拉赫曼·本·法伊萨尔大学）
University of Crete（克里特大学）；FORTH（希腊研究与技术基金会）；Athens University of Economics and Business（雅典经济与商业大学）
Deception detection is a task with many applications both in direct physical and in computer-mediated communication. Our focus is on automatic deception detection in text across cultures. We view culture through the prism of the individualism/collectivism dimension and we approximate culture by using country as a proxy. Having as a starting point recent conclusions drawn from the social psychology discipline, we explore if differences in the usage of specific linguistic features of deception across cultures can be confirmed and attributed to norms in respect to the individualism/collectivism d…
The Cultural Gene of Large Language Models: A Study on the Impact of Cross-Corpus Training on Model Values and Biases
Department of Communications（通讯系）；University of Malta（马耳他大学）；Faculty of Mathematics（数学系）；University of Primorska（普里摩尔卡大学）；Faculty of Electrical Engineering（电气工程系）；University of Montenegro（黑山大学）；Faculty of Science & Technology（科学与技术系）；University of the Faroe Islands（法罗群岛大学）；Department of Computer Science（计算机科学系）；San Francisco State University（旧金山州立大学）
Wharton School, University of Pennsylvania（宾夕法尼亚大学沃顿商学院）
Cultural items like songs have an important impact in creating and reinforcing stereotypes, biases, and discrimination. But the actual nature of such items is often less transparent. Take songs, for example. Are lyrics biased against women? And how have any such biases changed over time? Natural language processing of a quarter of a million songs over 50 years quantifies misogyny. Women are less likely to be associated with desirable traits (i.e., competence), and while this bias has decreased, it persists. Ancillary analyses further suggest that song lyrics may help drive shifts in societal s…
On the Systematic Challenges of Culturally Loaded Machine Translation: Dream of the Red Chamber as the Cultural Lens
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；School of Foreign Languages, East China University of Science and Technology（华东理工大学外国语学院）；Mejiro University（目白大学）
Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research
School of Computing, The University of Georgia（佐治亚大学计算机学院）；Department of Mathematical and Statistical Sciences, University of Alberta（阿尔伯塔大学数学与统计科学系）；University of California, Los Angeles（加州大学洛杉矶分校）；Department of Radiology, Massachusetts General Hospital and Harvard Medical School（麻省总医院放射科和哈佛医学院）
Disentangling Language and Culture for Evaluating Multilingual Large Language Models
Singapore Management University（新加坡管理学院）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；Institute of Trustworthy Embodied AI, Fudan University（复旦大学可信具身人工智能研究院）；DAMO Academy, Alibaba Group（阿里集团大模型实验室）；Singapore University of Technology and Design（新加坡科技设计大学）
