Translating culture-related content is vital for effective cross-cultural communication. However, many culture-specific items (CSIs) often lack viable translations across languages, making it challenging to collect high-quality, diverse parallel corpora with CSI annotations. This difficulty hinders the analysis of cultural awareness of machine translation (MT) systems, including traditional neural MT and the emerging MT paradigm using large language models (LLM). To address this gap, we introduce a novel parallel corpus, enriched with CSI annotations in 6 language pairs for investigating Culturally-Aware Machine Translation--CAMT. Furthermore, we design two evaluation metrics to assess CSI translations, focusing on their pragmatic translation quality. Our findings show the superior ability of LLMs over neural MTs in leveraging external cultural knowledge for translating CSIs, especially those lacking translations in the target culture.
From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation
Xinjiang Technical Institute of Physics \& Chemistry, Chinese Academy of Sciences Urumqi China；Xinjiang Technical Institute of Physics \& Chemistry, Chinese Academy of Sciences
University of Mannheim（曼海姆大学）；University of Technology Nuremberg（纽伦堡技术大学）；National Institute of Information and Communications Technology（信息与通信技术国家研究所）
DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Center for Language and Information Research, Wuhan University（武汉大学语言信息研究中心）；Jiangxi Normal University（江西师范大学）；The University of Manchester（曼彻斯特大学）；Yunnan Trrans Technology Co., Ltd.（云南翻译技术有限公司）；Malvern College Chengdu（成都马尔伯学院）
Social Bias in Multilingual Language Models: A Survey
School of Computer Science, University of Birmingham（伯明翰大学计算机科学学院）；Department of Information Systems and Computer Science, Ateneo de Manila University（马尼拉大学信息系统与计算机科学系）
Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China（计算与智能研究所，哈尔滨工业大学，深圳，中国）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室，深圳，中国）
CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding
Instituto Politécnico Nacional（墨西哥国立理工学院）；Haverford College（哈弗福德学院）；Wollo University（沃尔洛大学）；Pennsylvania State University（宾夕法尼亚州立大学）；University of Hamburg（汉堡大学）
The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models
University College London（伦敦大学学院）；Queen’s University Belfast（贝尔法斯特女王大学）；The University of Tokyo（东京大学）；University of Bisha（比沙大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；AI Centre, University College London（伦敦大学学院人工智能中心）
Probing Cultural Awareness in LLMs: A Case Study of Cross-Culture Aesthetic Stylistics
Department of Computing, Hong Kong Polytechnic University（香港理工大学计算机系）；Institute of Digital Twin, Eastern Institute of Technology, Ningbo（宁波东部技术研究所数字孪生研究所）；Department of Language Science and Technology, Hong Kong Polytechnic University（香港理工大学语言科学与技术系）；School of Design, Hong Kong Polytechnic University（香港理工大学设计学院）；Research Institute for Quantum Technology, Hong Kong Polytechnic University（香港理工大学量子技术研究所）；Department of Communication Science, Vrije Universiteit Amsterdam（阿姆斯特丹自由大学传播科学系）
MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西溪大学工程学院）；Southern University of Science and Technology（南方科技大学）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西溪研究院先进技术研究所）
Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Georgia Institute of Technology（佐治亚理工学院）；Samsung R&D Institute Philippines（三星菲律宾研发院）；Sungkyunkwan University（成均馆大学）；Tohoku University（东北大学）；National University of Singapore（新加坡国立大学）；University of Copenhagen（哥本哈根大学）；University of Michigan（密歇根大学）；Indian Institute of Technology Delhi（印度理工学院德里）；Institute of Science Tokyo（东京科学研究院）
WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
University of International Relations（国际关系学院）；Kedge Business School（凯致商学院）；Peking University（北京大学）；Tsinghua University（清华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Capital Normal University（首都师范大学）
MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
Islamic University of Technology（伊斯兰科技大学）；South East University（东南大学）；Vector Institute（矢量研究所）；University of British Columbia（不列颠哥伦比亚大学）；York University（约克大学）
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Alibaba Group（阿里巴巴集团）；Beijing Language and Culture University（北京语言大学）；LMU Munich（慕尼黑大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）；IBM Research AI, UAE（阿联酋IBM人工智能研究）；MBZUAI；Harbin Institute of Technology, Shenzhen（深圳哈尔滨工业大学）；Southern University of Science and Technology（南方科技大学）
XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad
Department of Computer Science, National Centre for Text Mining, The University of Manchester（计算机科学系，国家文本挖掘中心，曼彻斯特大学）；ELLIS Manchester（曼彻斯特ELLIS）；School of Computing, Queen’s University, Ontario, Canada（计算学院，加拿大皇后大学）；Computer Science, University of Illinois Chicago（计算机科学，伊利诺伊大学芝加哥分校）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）；Department of Computer Science and Artificial Intelligence, Umm Al-Qura University, Makkah, Saudi Arabia（计算机科学与人工智能系，乌姆·阿勒·卡拉大学，麦加，沙特阿拉伯）
Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness
Division of Computational Health Sciences, Department of Surgery University of Minnesota（大学医学计算科学系，外科系）；Institute for Health Informatics University of Minnesota（健康信息学研究所，明尼苏达大学）；Department of Electrical and Computer Engineering University of Minnesota（电气与计算机工程系，明尼苏达大学）
