MBBQ：生成式大语言模型中刻板印象的跨语言比较数据集
MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs
AI总结：
提出MBBQ多语言问答偏见基准，扩展至荷兰语、西班牙语和土耳其语，验证了非英语语言中LLMs的刻板印象偏见更严重，并发现跨语言偏见差异显著。
摘要：
生成式大语言模型（LLMs）已被证明会表现出有害的偏见和刻板印象。虽然安全微调通常以英语进行（如果进行的话），但这些模型被许多不同语言的用户使用。已有证据表明，这些模型的性能在不同语言间不一致，并且它们会根据用户的人口统计学特征进行歧视。受此启发，我们研究了LLMs表现出的社会刻板印象是否因提示所用的语言而异，同时控制文化差异和任务准确性。为此，我们提出了MBBQ（多语言问答偏见基准），这是英文BBQ数据集的一个精心策划的扩展版本，扩展至荷兰语、西班牙语和土耳其语，用于衡量这些语言中普遍存在的刻板印象。我们进一步为MBBQ补充了一个平行的控制数据集，以独立于偏见衡量问答任务的任务性能。我们基于多个开源和专有LLMs的结果证实，即使控制了文化差异，一些非英语语言的偏见比英语更严重。此外，我们观察到除最准确的模型外，所有模型在偏见行为上均存在显著的跨语言差异。通过发布MBBQ，我们希望鼓励在多语言环境中对偏见进行进一步研究。数据集和代码可在https://github.com/Veranep/MBBQ获取。
英文摘要：
Generative large language models (LLMs) have been shown to exhibit harmful biases and stereotypes. While safety fine-tuning typically takes place in English, if at all, these models are being used by speakers of many different languages. There is existing evidence that the performance of these models is inconsistent across languages and that they discriminate based on demographic factors of the user. Motivated by this, we investigate whether the social stereotypes exhibited by LLMs differ as a function of the language used to prompt them, while controlling for cultural differences and task accuracy. To this end, we present MBBQ (Multilingual Bias Benchmark for Question-answering), a carefully curated version of the English BBQ dataset extended to Dutch, Spanish, and Turkish, which measures stereotypes commonly held across these languages. We further complement MBBQ with a parallel control dataset to measure task performance on the question-answering task independently of bias. Our results based on several open-source and proprietary LLMs confirm that some non-English languages suffer from bias more than English, even when controlling for cultural shifts. Moreover, we observe significant cross-lingual differences in bias behaviour for all except the most accurate models. With the release of MBBQ, we hope to encourage further research on bias in multilingual settings. The dataset and code are available at https://github.com/Veranep/MBBQ.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
