Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how well these models perform on topics relevant to other cultures, whose presence on the web is not that prominent. To address this gap, we introduce BertaQA, a multiple-choice trivia dataset that is parallel in English and Basque. The dataset consists of a local subset with questions pertinent to the Basque culture, and a global subset with questions of broader interest. We find that state-of-the-art LLMs struggle with local cultural knowledge, even as they excel on global topics. However, we show that continued pre-training in Basque significantly improves the models' performance on Basque culture, even when queried in English. To our knowledge, this is the first solid evidence of knowledge transfer from a low-resource to a high-resource language. Our analysis sheds light on the complex interplay between language and knowledge, and reveals that some prior findings do not fully hold when reassessed on local topics. Our dataset and evaluation code are available under open licenses at https://github.com/juletx/BertaQA.
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
KAIST(韩国科学技术院);Cardiff University(卡迪夫大学);Universität Hamburg(汉堡大学);Bahir Dar University(巴赫达尔大学);NAVER AI Lab(NAVER人工智能实验室);Imperial College London(伦敦帝国学院)
RiverEcho: Real-Time Interactive Digital System for Ancient Yellow River Culture
School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);School of Computer Science, Peking University(北京大学计算机学院);The School of History, Renmin University of China(中国人民大学历史学院);Pengcheng Laboratory(鹏城实验室);School of Arts, Peking University(北京大学艺术学院);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系)
Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement
Department of Computer Science and Engineering, Dongguk University(计算机科学与工程系,东国大学);Department of Computer Science and Artificial Intelligence, Dongguk University(计算机科学与人工智能系,东国大学)
Worlds Within Words: Translating Culture in Ancient Chinese Texts with Multi-Agent Coordination
NLP2 CT Lab, Department of Computer and Information Science, University of Macau(澳门大学自然语言处理与计算机实验室,计算机与信息科学系);Institute of International Language Services Studies, Macau Millennium College(澳门 millennium 学院国际语言服务研究学院)
CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming
University of Washington(华盛顿大学);University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(人工智能向量研究所);McGill University(麦吉尔大学);University of Copenhagen(哥本哈根大学);Stanford University(斯坦福大学)
DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context
Natural Language and Information Processing Lab (NLIP) Indian Institute of Technology Hyderabad(自然语言与信息处理实验室(NLIP)印度理工学院海得拉巴)
CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs
Hefei University of Technology(合肥工业大学);Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China(民族语言智能分析与安全治理国家重点实验室,中央民族大学);School of Information Engineering, Minzu University of China(中央民族大学信息工程学院)
Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditions
Amar Nath and Shashi Khosla School of Information Technology, Indian Institute of Technology Delhi(阿玛纳特和沙希·科斯拉信息科技学院,印度理工学院德里);Computer Science and Engineering Department, Delhi Technological University(德里技术大学计算机科学与工程系);Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur(印度理工学院Khargpur计算机科学与工程系)
LLMs as Cultural Archives: Cultural Commonsense Knowledge Graph Extraction
Mohamed bin Zayed University of Artificial Intelligence(莫扎德·本·扎耶德人工智能大学);Ubiquitous Knowledge Processing Lab (UKP Lab)(无处不在知识处理实验室);Department of Computer Science(计算机科学系);Hessian Center for AI (hessian.AI)(黑森人工智能中心);Technische Universität Darmstadt(德累斯顿技术大学)
SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture
Department of Computer Science and Engineering, Indian Institute of Technology Patna(印度理工学院帕纳布分校计算机科学与工程系);Department of Computer Science, Banasthali Vidyapeeth University(班斯塔利大学计算机科学系)
Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
United International University(国际联合大学);BRAC University(布拉塔克大学);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校)