WinoQueer-NL：评估荷兰语模型对LGBTQ+身份的偏见
WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identities
浏览论文内容
中文总结 AI 辅助
本研究开发文化适配的荷兰语数据集WinoQueer-NL，评估荷兰语MLMs和ARLMs对LGBTQ+身份的偏见，发现模型对跨性别等身份偏见显著，凸显文化相关数据集的重要性。
中文摘要 AI 辅助
尽管英语语言模型的反酷儿偏见已被广泛研究，但荷兰语模型仍未得到充分研究。为解决这一空白，我们基于英语WinoQueer基准开发了文化和语言适配的荷兰语数据集，包含刻板印象和反刻板印象的句子对。为验证并扩展该数据集，我们对43名荷兰酷儿参与者进行了在线调查，确认171个刻板印象中有145个具有文化相关性，并通过自由文本回复识别出22种新偏见。最终发布的数据集包含42906个句子，使用一系列荷兰语特定模型和多语言模型进行评估，包括掩码语言模型（MLMs）和自回归语言模型（ARLMs），偏见通过比较刻板印象与反刻板印象句子对数似然的分数来测量。尽管模型的平均偏见分数看似中立（约50%），但更深入分析显示存在显著差异：部分模型对跨性别身份的刻板印象句子的偏好高达97%，而对同性恋相关对的偏好仅为6%，跨性别和非二元身份始终获得最高偏见分数。我们的研究结果强调了基于文化的数据集对于评估和减轻荷兰语模型中对边缘群体产生不成比例影响的偏见的重要性。
英文摘要
While English language models have been widely examined for anti-queer bias, Dutch models remain understudied. To address this gap, we developed a culturally and linguistically adapted Dutch dataset based on the English WinoQueer benchmark, containing pairs of stereotypical and counter-stereotypical sentences. To validate and expand it, we conducted an online survey with 43 Dutch queer participants, confirming 145 of 171 stereotypes as culturally relevant and identifying 22 new biases through free-text responses. The final released dataset, comprising 42,906 sentences, was evaluated using a range of Dutch-specific and multilingual models, including both masked language models (MLMs) and autoregressive language models (ARLMs), with bias measured via a score comparing log-likelihoods of stereotypical versus counter-stereotypical sentences. While the mean bias score across models appeared neutral (~50%), closer analysis revealed significant disparities: some models favored stereotypical sentences up to 97% of the time for transgender identities, but only 6% of the time for gay-related pairs, with transgender and non-binary identities consistently receiving the highest bias scores. Our findings highlight the importance of culturally grounded datasets for evaluating and mitigating biases that disproportionately impact marginalized groups in Dutch language models.
发表机构
- Maastricht University（马斯特里赫特大学）
机构由 AI 辅助整理，请以论文原文为准。
