我们呈现了一项对90余篇近期论文的综述,这些论文旨在研究大语言模型(LLMs)中的文化表征与包容性。我们观察到,没有一项研究明确地定义“文化”,这是一个复杂、多面向的概念;相反,它们在一些专门设计的数据集上探测模型,这些数据集代表了“文化”的某些方面。我们将这些方面称为文化的代理(proxies),并沿着人口统计学代理和语义代理两个维度对其进行组织。我们还对所使用的探测方法进行了分类。我们的分析表明,只有“文化”的某些方面,如价值观和目标,得到了研究,而其他几个有趣且重要的方面,尤其是众多的语义领域(Thompson et al., 2020)和aboutness(Hershcovich et al., 2022),尚未被探索。另外两个关键空白是探测技术缺乏鲁棒性,以及缺乏关于基于LLM的应用中文化误表征和表征不足所产生影响的情境化研究。
英文摘要:
We present a survey of more than 90 recent papers that aim to study cultural representation and inclusion in large language models (LLMs). We observe that none of the studies explicitly define "culture, which is a complex, multifaceted concept; instead, they probe the models on some specially designed datasets which represent certain aspects of "culture". We call these aspects the proxies of culture, and organize them across two dimensions of demographic and semantic proxies. We also categorize the probing methods employed. Our analysis indicates that only certain aspects of ``culture,'' such as values and objectives, have been studied, leaving several other interesting and important facets, especially the multitude of semantic domains (Thompson et al., 2020) and aboutness (Hershcovich et al., 2022), unexplored. Two other crucial gaps are the lack of robustness of probing techniques and situated studies on the impact of cultural mis- and under-representation in LLM-based applications.
XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad
Department of Computer Science, National Centre for Text Mining, The University of Manchester(计算机科学系,国家文本挖掘中心,曼彻斯特大学);ELLIS Manchester(曼彻斯特ELLIS);School of Computing, Queen’s University, Ontario, Canada(计算学院,加拿大皇后大学);Computer Science, University of Illinois Chicago(计算机科学,伊利诺伊大学芝加哥分校);ELLIS Institute Finland(芬兰ELLIS研究所);University of Turku(图尔库大学);Department of Computer Science and Artificial Intelligence, Umm Al-Qura University, Makkah, Saudi Arabia(计算机科学与人工智能系,乌姆·阿勒·卡拉大学,麦加,沙特阿拉伯)
Do LLMs Understand Wine Descriptors Across Cultures? A Benchmark for Cultural Adaptations of Wine Reviews
Department of Computer Science, University of Copenhagen(哥本哈根大学计算机科学系);Department of Food and Resource Economics, University of Copenhagen(哥本哈根大学食品与资源经济学系);Department of Food Science, University of Copenhagen(哥本哈根大学食品科学系);Department of Computer Science, Aarhus University(阿arhus大学计算机科学系)
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
KAIST(韩国科学技术院);Cardiff University(卡迪夫大学);Universität Hamburg(汉堡大学);Bahir Dar University(巴赫达尔大学);NAVER AI Lab(NAVER人工智能实验室);Imperial College London(伦敦帝国学院)
DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context
Natural Language and Information Processing Lab (NLIP) Indian Institute of Technology Hyderabad(自然语言与信息处理实验室(NLIP)印度理工学院海得拉巴)
The Curious Case of Curiosity across Human Cultures and LLMs
University of Michigan - Ann Arbor(密歇根大学安阿伯分校);University of Toronto(多伦多大学);Vector Institute(向量研究所);MPI for Intelligent Systems, Tubingen, Germany(图宾根德国智能系统研究所)
Is It Bad to Work All the Time? Cross-Cultural Evaluation of Social Norm Biases in GPT-4
Department of Computer Science, University of British Columbia(不列颠哥伦比亚大学计算机科学系);Vector Institute for AI(人工智能矢量研究所);Department of Linguistics, University of British Columbia(不列颠哥伦比亚大学语言学系);Mila – Quebec AI Institute(魁北克人工智能研究所);School of Computer Science, McGill University(麦吉尔大学计算机科学学院);Department of Sociology, University of British Columbia(不列颠哥伦比亚大学社会学系)
On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures
Johannes Gutenberg University Mainz(明斯特-雅各布大学);University of Mannheim(曼海姆大学);Center For Artificial Intelligence and Data Science, University of Würzburg(乌尔姆大学人工智能与数据科学中心);University of Colorado Boulder(科罗拉多大学博尔德分校)
Beyond Instrumental and Substitutive Paradigms: Introducing Machine Culture as an Emergent Phenomenon in Large Language Models
Institute of Neuroscience, Chinese Academy of Sciences(中国科学院神经科学研究所);University of Cambridge(剑桥大学);Chinese University of Hong Kong(香港中文大学);Institute of Public Opinion Survey, Hong Kong SAR(香港特别行政区民意调查研究所);Tsinghua University(清华大学)
Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditions
Amar Nath and Shashi Khosla School of Information Technology, Indian Institute of Technology Delhi(阿玛纳特和沙希·科斯拉信息科技学院,印度理工学院德里);Computer Science and Engineering Department, Delhi Technological University(德里技术大学计算机科学与工程系);Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur(印度理工学院Khargpur计算机科学与工程系)
Probing Cultural Awareness in LLMs: A Case Study of Cross-Culture Aesthetic Stylistics
Department of Computing, Hong Kong Polytechnic University(香港理工大学计算机系);Institute of Digital Twin, Eastern Institute of Technology, Ningbo(宁波东部技术研究所数字孪生研究所);Department of Language Science and Technology, Hong Kong Polytechnic University(香港理工大学语言科学与技术系);School of Design, Hong Kong Polytechnic University(香港理工大学设计学院);Research Institute for Quantum Technology, Hong Kong Polytechnic University(香港理工大学量子技术研究所);Department of Communication Science, Vrije Universiteit Amsterdam(阿姆斯特丹自由大学传播科学系)
Research Borderlands: Analysing Writing Across Research Cultures
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校Siebel计算与数据科学学院);Pioneer Centre for AI, University of Copenhagen(哥本哈根大学先锋人工智能中心);Allen Institute for Artificial Intelligence (Ai2)(人工智能研究院)
CulturePark: Boosting Cross-cultural Understanding in Large Language Models
Institute of Software, CAS(中国科学院软件研究所);Idiap Research Institute(Idiap研究所);Westlake University(西湖大学);Squirrel AI(松鼠AI);Microsoft Research(微软研究院);William & Mary(威廉玛丽学院)
CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding
Instituto Politécnico Nacional(墨西哥国立理工学院);Haverford College(哈弗福德学院);Wollo University(沃尔洛大学);Pennsylvania State University(宾夕法尼亚州立大学);University of Hamburg(汉堡大学)
Tailored Emotional LLM-Supporter: Enhancing Cultural Sensitivity
Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technische Universität Darmstadt(通用知识处理实验室(UKP实验室)计算机科学系和海斯曼人工智能中心(hessian.AI)德意志联邦人家电大学);Department of Psychology, Bar-Ilan University(心理学系,巴伊兰大学)