arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.12464cs.CL
收藏

NormAd:衡量大语言模型文化适应性的框架

NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models

Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)

Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, Maarten Sap

AI总结:

NormAd框架评估大语言模型的文化适应性,通过基准NormAd-Eti(含75国2600个情境)发现模型判断社会可接受性能力弱于人类,且对英语文化适应性更强。

摘要:

为了有效且安全地部署给全球用户群体,大语言模型(LLMs)可能不仅需要了解用户价值观和文化,还需要调整其输出以适应这些价值观和文化。我们引入了NormAd,一个评估框架,用于衡量LLMs的文化适应性,具体衡量它们在不同文化规范具体性水平(从抽象价值观到明确社会规范)下判断社会可接受性的能力。作为我们框架的一个实例,我们创建了NormAd-Eti,一个包含来自75个国家的2600个情境描述的基准数据集,这些情境涉及社交礼仪相关的文化规范。通过在NormAd-Eti上的全面实验,我们发现LLMs在判断这些不同文化背景下的社会可接受性方面存在困难,并且对以英语为中心的文化表现出比全球南方文化更强的适应性。即使在提供相关社会规范的最简单设置中,最好的LLMs的性能(<82%)也落后于人类(>95%)。在提供抽象价值观和国家信息的设置中,模型性能大幅下降(<60%),而人类的准确率仍然很高(>90%)。此外,我们发现模型更擅长识别社会可接受的情境,而不是不可接受的情境。我们的研究结果展示了LLMs在社会文化推理方面的当前缺陷,这些缺陷阻碍了它们对全球受众的适应性。

英文摘要:

To be effectively and safely deployed to global user populations, large language models (LLMs) may need to adapt outputs to user values and cultures, not just know about them. We introduce NormAd, an evaluation framework to assess LLMs' cultural adaptability, specifically measuring their ability to judge social acceptability across varying levels of cultural norm specificity, from abstract values to explicit social norms. As an instantiation of our framework, we create NormAd-Eti, a benchmark of 2.6k situational descriptions representing social-etiquette related cultural norms from 75 countries. Through comprehensive experiments on NormAd-Eti, we find that LLMs struggle to accurately judge social acceptability across these varying degrees of cultural contexts and show stronger adaptability to English-centric cultures over those from the Global South. Even in the simplest setting where the relevant social norms are provided, the best LLMs' performance (< 82\%) lags behind humans (> 95\%). In settings with abstract values and country information, model performance drops substantially (< 60\%), while human accuracy remains high (> 90\%). Furthermore, we find that models are better at recognizing socially acceptable versus unacceptable situations. Our findings showcase the current pitfalls in socio-cultural reasoning of LLMs which hinder their adaptability for global audiences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑