Large Language Models (LLMs), such as ChatGPT, are widely used to generate content for various purposes and audiences. However, these models may not reflect the cultural and emotional diversity of their users, especially for low-resource languages. In this paper, we investigate how ChatGPT represents Hausa's culture and emotions. We compare responses generated by ChatGPT with those provided by native Hausa speakers on 37 culturally relevant questions. We conducted experiments using emotion analysis and applied two similarity metrics to measure the alignment between human and ChatGPT responses. We also collected human participants ratings and feedback on ChatGPT responses. Our results show that ChatGPT has some level of similarity to human responses, but also exhibits some gaps and biases in its knowledge and awareness of the Hausa culture and emotions. We discuss the implications and limitations of our methodology and analysis and suggest ways to improve the performance and evaluation of LLMs for low-resource languages.
Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports
Indian Institute of Technology Patna（印度理工学院帕纳布分校）；Sardar Patel Institute of Technology（萨达尔·帕特尔技术学院）；Universitas Gadjah Mada（加查马大学）；King Mongkut’s Institute of Technology Ladkrabang（拉差班国王技术学院）；Shenzhen Technology University（深圳技术大学）；Université de Toulouse（图卢兹大学）
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
KAIST（韩国科学技术院）；Cardiff University（卡迪夫大学）；Universität Hamburg（汉堡大学）；Bahir Dar University（巴赫达尔大学）；NAVER AI Lab（NAVER人工智能实验室）；Imperial College London（伦敦帝国学院）
AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic
Saarland University（萨尔兰大学）；AIMS AMMI；Resonance AI4D Lab（Resonance AI4D实验室）；Addis Ababa University（亚的斯亚贝巴大学）；HILCOE School of Computer Science and Technology（HILCOE计算机科学与技术学院）；University of Hamburg（汉堡大学）
University of Copenhagen（哥本哈根大学）；University of Trento（特伦托大学）；Uppsala University（乌普萨拉大学）；KU Leuven（鲁汶大学）；University of Sheffield（谢菲尔德大学）
Various efforts in the Natural Language Processing (NLP) community have been made to accommodate linguistic diversity and serve speakers of many different languages. However, it is important to acknowledge that speakers and the content they produce and require, vary not just by language, but also by culture. Although language and culture are tightly linked, there are important differences. Analogous to cross-lingual and multilingual NLP, cross-cultural and multicultural NLP considers these differences in order to better serve users of NLP systems. We propose a principled framework to frame the…
VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding
DJCAD, University of Dundee, United Kingdom（邓迪大学DJCAD部门）；University of California, Santa Cruz, USA（加州大学圣克ruz分校）；Analogy AI；East China Normal University, China（华东师范大学）；Nanjing University, China（南京大学）；Department of Mechanical Engineering, School of Engineering, University of Birmingham（伯明翰大学工程学院机械工程系）
Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12
INESC-ID；Instituto Superior Técnico - University of Lisbon（理工学院 - 里斯本大学）；Department of Technology, Operations, and Statistics, New York University（技术、运营与统计系，纽约大学）；Speech Technology and Machine Learning Group - Universidad Politécnica de Madrid（语音技术与机器学习小组 - 马德里理工大学）；Carnegie Mellon University（卡内基梅隆大学）
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
Capital One；SEACrowd；NAIST；ITB；UofT；KAIST；JAIST；Cardiff University；Independent；HKUST；SMU；Masakhane；University of Lagos；HK PolyU；Boston University；Columbia University；MBZUAI；Monash University；UCSB；Tokyo Tech；AI Singapore；NICT；Cohere；McGill；MILA；Canada CIFAR AI Chair；Ontario Tech
Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability
Google DeepMind（谷歌DeepMind）；University of Alabama in Huntsville（阿拉巴马大学亨茨维尔分校）；Google（谷歌）；University of Missouri Columbia（密苏里大学哥伦比亚分校）；Chunghwa Telecom Laboratories（春木电信实验室）；University of Southern California（南加州大学）；Polytechnique Montreal（蒙特利尔理工学院）；Nutanix；ThinkEvolve Labs（ThinkEvolve实验室）；UCL（伦敦大学学院）；Infocomm Media Development Authority（信息通信媒体发展局）；Monark Health（Monark健康）；Seoul National University（首尔国立大学）；Microsoft（微软）；Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), Indian Institute of Technology Madras（负责任人工智能中心（CeRAI）、瓦达威人工智能学校（WSAI）、印度理工学院马德拉斯分校）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Microsoft Research India（微软印度研究院）；Stanford University（斯坦福大学）；Argonne National Laboratory（阿贡国家实验室）；University of Oxford（牛津大学）；KAIST（韩国科学技术院）；Yonsei University（延世大学）；Amazon（亚马逊）；UIUC（伊利诺伊大学香槟分校）；Rochester Institute of Technology（罗切斯特理工学院）；Xenoscube Inc.（Xenoscube公司）；Korea AI Safety Institute (K-AISI)（韩国人工智能安全研究所（K-AISI））；MLCommons；CommonGround；Artifex Labs（Artifex实验室）
University of Cambridge（剑桥大学）；University of Copenhagen（哥本哈根大学）；Mila – Quebec Artificial Intelligence Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）
The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet. While one can hardly overestimate how much this benchmark contributed to progress in computer vision, it is mostly derived from lexical databases and image queries in English, resulting in source material with a North American or Western European bias. Therefore, we devise a new protocol to construct an ImageNet-style hierarchy representative of more languages and cultures. In particular, we let the selection of both concepts and images…
