WorldValuesBench：面向语言模型多元文化价值意识的大规模基准数据集
WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models
AI总结：
本文提出大规模基准数据集WorldValuesBench，基于WVS构建超2000万样本以评估语言模型多元文化价值预测能力，案例研究表明现有强模型在该任务上表现仍具挑战性。
摘要：
多元文化人类价值意识对于语言模型（LMs）生成安全且个性化响应的能力至关重要。然而，由于计算机科学界缺乏关于多元文化价值的大规模真实世界数据，LMs的这种意识尚未得到充分研究。本文提出了WorldValuesBench，一个面向多元文化价值预测任务的全球多样化大规模基准数据集，该任务要求模型根据人口统计背景对价值问题生成评分响应。我们的数据集源自一项具有影响力的社会科学项目World Values Survey（WVS），该项目收集了全球94,728名参与者对数百个价值问题（例如社会、经济、伦理问题）的回答。我们已从WVS响应中构建了超过2000万个“(人口统计属性, 价值问题) $\ ightarrow$ 回答”类型的样本。我们使用该数据集进行了案例研究，结果表明该任务对强大的开源和闭源模型均具有挑战性。仅在 $11.1\%$、$25.0\%$、$72.2\%$ 和 $75.0\%$ 的问题上，Alpaca-7B、Vicuna-7B-v1.5、Mixtral-8x7B-Instruct-v0.1 和 GPT-3.5 Turbo 才能分别实现与人类归一化答案分布小于 $0.2$ 的 Wasserstein 1-距离。WorldValuesBench 为研究 LMs 在多元文化价值意识方面的局限性与机遇开辟了新的研究途径。
英文摘要：
The awareness of multi-cultural human values is critical to the ability of language models (LMs) to generate safe and personalized responses. However, this awareness of LMs has been insufficiently studied, since the computer science community lacks access to the large-scale real-world data about multi-cultural values. In this paper, we present WorldValuesBench, a globally diverse, large-scale benchmark dataset for the multi-cultural value prediction task, which requires a model to generate a rating response to a value question based on demographic contexts. Our dataset is derived from an influential social science project, World Values Survey (WVS), that has collected answers to hundreds of value questions (e.g., social, economic, ethical) from 94,728 participants worldwide. We have constructed more than 20 million examples of the type "(demographic attributes, value question) $\rightarrow$ answer" from the WVS responses. We perform a case study using our dataset and show that the task is challenging for strong open and closed-source models. On merely $11.1\%$, $25.0\%$, $72.2\%$, and $75.0\%$ of the questions, Alpaca-7B, Vicuna-7B-v1.5, Mixtral-8x7B-Instruct-v0.1, and GPT-3.5 Turbo can respectively achieve $<0.2$ Wasserstein 1-distance from the human normalized answer distributions. WorldValuesBench opens up new research avenues in studying limitations and opportunities in multi-cultural value awareness of LMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
