NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, Maarten Sap
备注:
Published at NAACL 2025, Albuquerque, New Mexico, USA
发表信息:
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) 2373-2403
To be effectively and safely deployed to global user populations, large language models (LLMs) may need to adapt outputs to user values and cultures, not just know about them. We introduce NormAd, an evaluation framework to assess LLMs' cultural adaptability, specifically measuring their ability to judge social acceptability across varying levels of cultural norm specificity, from abstract values to explicit social norms. As an instantiation of our framework, we create NormAd-Eti, a benchmark of 2.6k situational descriptions representing social-etiquette related cultural norms from 75 countries. Through comprehensive experiments on NormAd-Eti, we find that LLMs struggle to accurately judge social acceptability across these varying degrees of cultural contexts and show stronger adaptability to English-centric cultures over those from the Global South. Even in the simplest setting where the relevant social norms are provided, the best LLMs' performance (< 82\%) lags behind humans (> 95\%). In settings with abstract values and country information, model performance drops substantially (< 60\%), while human accuracy remains high (> 90\%). Furthermore, we find that models are better at recognizing socially acceptable versus unacceptable situations. Our findings showcase the current pitfalls in socio-cultural reasoning of LLMs which hinder their adaptability for global audiences.
Whispers of Many Shores: Cultural Alignment through Collaborative Cultural Expertise
Arizona State University(亚利桑那州立大学);National Taiwan University(台湾国立大学);Carnegie Mellon University(卡内基梅隆大学);New York University(纽约大学);Indian Institute of Technology Hyderabad(印度海得拉巴理工学院);Minitab(Minitab公司);Trinity College Dublin(都柏林圣三一学院)
The Curious Case of Curiosity across Human Cultures and LLMs
University of Michigan - Ann Arbor(密歇根大学安阿伯分校);University of Toronto(多伦多大学);Vector Institute(向量研究所);MPI for Intelligent Systems, Tubingen, Germany(图宾根德国智能系统研究所)
Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research
School of Computing, The University of Georgia(佐治亚大学计算机学院);Department of Mathematical and Statistical Sciences, University of Alberta(阿尔伯塔大学数学与统计科学系);University of California, Los Angeles(加州大学洛杉矶分校);Department of Radiology, Massachusetts General Hospital and Harvard Medical School(麻省总医院放射科和哈佛医学院)
Analyzing Cognitive Differences Among Large Language Models through the Lens of Social Worldview
Wangxuan Institute of Computer Technology, Peking University(北京大学计算机技术研究院);Information Management Department, Peking University(北京大学信息管理系);Renmin University of China(中国人民大学)
When LLMs Team Up: The Emergence of Collaborative Affective Computing
Department of Computing, Hong Kong Polytechnic University(香港理工大学计算机系);School of Data Science, Lingnan University(岭南大学数据科学学院);School of Computer Science and the Data Science Institute, University of Technology Sydney(悉尼大学计算机科学学院和数据科学研究所);Education University of Hong Kong(香港教育大学)
Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach
Isfahan University of Technology(伊斯法罕理工大学);Shahid Bahonar University of Kerman(克尔曼沙希德·巴霍纳尔大学);University of Tehran(德黑兰大学);Hunter College, City University of New York(纽约市立大学亨特学院)
Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Models
Research Center Trustworthy Data Science and Security of the University Alliance Ruhr(鲁尔大学联盟可信数据科学与安全研究中心);Faculty of Informatics, Technical University Dortmund(多特蒙德工业大学信息学院)