The cultural landscape of interactions with dialogue agents is a compelling yet relatively unexplored territory. It's clear that various sociocultural aspects -- from communication styles and beliefs to shared metaphors and knowledge -- profoundly impact these interactions. To delve deeper into this dynamic, we introduce cuDialog, a first-of-its-kind benchmark for dialogue generation with a cultural lens. We also develop baseline models capable of extracting cultural attributes from dialogue exchanges, with the goal of enhancing the predictive accuracy and quality of dialogue agents. To effectively co-learn cultural understanding and multi-turn dialogue predictions, we propose to incorporate cultural dimensions with dialogue encoding features. Our experimental findings highlight that incorporating cultural value surveys boosts alignment with references and cultural markers, demonstrating its considerable influence on personalization and dialogue quality. To facilitate further exploration in this exciting domain, we publish our benchmark publicly accessible at https://github.com/yongcaoplus/cuDialog.
SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture
Department of Computer Science and Engineering, Indian Institute of Technology Patna(印度理工学院帕纳布分校计算机科学与工程系);Department of Computer Science, Banasthali Vidyapeeth University(班斯塔利大学计算机科学系)
Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset
The University of British Columbia(不列颠哥伦比亚大学);Prince Sultan University(苏丹亲王大学);Tuwaiq Academy(图瓦伊克学院);Hadhramout University(哈德拉毛大学);Misr University for Science & Technology(埃及科技大学);KFUPM(法赫德国王石油与矿业大学);Taibah University(塔伊巴大学);WMU(西密歇根大学);UM6P(穆罕默德六世理工大学);IBB University(伊卜大学);Cairo University(开罗大学);UCA(卡迪·阿亚德大学);Qafza Tech(卡夫扎科技公司);JUST(约旦科技大学);KAUST(阿卜杜拉国王科技大学);Damascus University(大马士革大学);University of Nouakchott(努瓦克肖特大学);Birzeit University(比尔宰特大学);Imam Abdulrahman Bin Faisal University(伊玛目阿卜杜拉赫曼·本·费萨尔大学);Hamad Bin Khalifa University(哈马德·本·哈利法大学);Invertible AI(Invertible AI 公司)
该研究构建覆盖全部阿拉伯国家、含逾30.9万样本的阿拉伯语多模态文化数据集 PEARL 及 PEARL-LITE、PEARL-X,验证推理中心指令对齐可提升 LVLM 文化理解。
Exploring Cultural Nuances in Emotion Perception Across 15 African Languages
Northeastern University(东北大学);Bayero University Kano(卡诺州巴耶罗大学);Instituto Politécnico Nacional(墨西哥国立理工学院);Wollo University(沃洛大学);University of Pretoria(比勒陀利亚大学);University of Hamburg(汉堡大学);Imperial College London(伦敦帝国学院)
HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMs
Kyushu University(九州大学);hon9kon9ize;The Education University of Hong Kong(香港教育大学);The University of Hong Kong(香港大学);Independent Researcher(独立研究者);Votee AI;Beever AI
MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);East China Normal University(东华大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Institute of High Performance Computing, A*STAR(高性能计算研究所,A*STAR)
Do LLMs Understand Wine Descriptors Across Cultures? A Benchmark for Cultural Adaptations of Wine Reviews
Department of Computer Science, University of Copenhagen(哥本哈根大学计算机科学系);Department of Food and Resource Economics, University of Copenhagen(哥本哈根大学食品与资源经济学系);Department of Food Science, University of Copenhagen(哥本哈根大学食品科学系);Department of Computer Science, Aarhus University(阿arhus大学计算机科学系)
CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding
Instituto Politécnico Nacional(墨西哥国立理工学院);Haverford College(哈弗福德学院);Wollo University(沃尔洛大学);Pennsylvania State University(宾夕法尼亚州立大学);University of Hamburg(汉堡大学)
Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs
Department of Computer Science, National Center for Text Mining, The University of Manchester(计算机科学系,文本挖掘国家中心,曼彻斯特大学);Department of Computer Science and Engineering, Islamic University of Technology(计算机科学与工程系,伊斯兰技术大学)
From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation
Xinjiang Technical Institute of Physics \& Chemistry, Chinese Academy of Sciences Urumqi China;Xinjiang Technical Institute of Physics \& Chemistry, Chinese Academy of Sciences
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
AIM Intelligence(AIM智能研究院);Microsoft(微软公司);Korea AISI(韩国人工智能研究所);KT Corporation(KT公司);BMW Group(宝马集团);Coinbase(Coinbase公司);Technical University of Munich(慕尼黑技术大学);Ankara University(安卡拉大学);Cyril Amarchand Mangaldas(Cyril Amarchand Mangaldas法律事务所);Seoul National University(首尔国立大学)