AfriStereo：一种基于文化背景的数据集，用于评估大型语言模型中的刻板印象偏见
AfriStereo: A Culturally Grounded Dataset for Evaluating Stereotypical Bias in Large Language Models
AI总结：
AfriStereo通过构建基于非洲文化的数据集，评估大型语言模型中的刻板印象偏见，揭示了模型在性别、年龄、职业等维度上的系统性偏见。
摘要：
现有的AI偏见评估基准大多反映了西方观点，导致非洲语境代表性不足，并在各种领域中允许有害的刻板印象。为解决这一差距，我们引入了AfriStereo，这是第一个开源的非洲刻板印象数据集和评估框架，扎根于当地的 socio-cultural 语境。通过在塞内加尔、肯尼亚和尼日利亚进行社区参与的努力，我们收集了1,163个刻板印象，涵盖性别、种族、宗教、年龄和职业。通过少量示例提示与人机协同验证，我们扩展了数据集，使其超过5,000个刻板印象-反刻板印象对。条目通过语义聚类和由文化知情审查员手动注释进行验证。初步评估语言模型显示，11个模型中有9个表现出统计显著的偏见，偏见偏好比率（BPR）在0.63到0.78之间（p <= 0.05），表明系统偏好刻板印象而非反刻板印象，特别是在年龄、职业和性别维度上。领域特定模型在我们的设置中似乎表现出较弱的偏见，这表明任务特定的训练可能减轻一些关联。展望未来，AfriStereo为未来基于文化背景的偏见评估和缓解研究开辟了道路，为AI社区提供了关键的方法，以构建更加公平、情境感知和全球包容的NLP技术。
英文摘要：
Existing AI bias evaluation benchmarks largely reflect Western perspectives, leaving African contexts underrepresented and enabling harmful stereotypes in applications across various domains. To address this gap, we introduce AfriStereo, the first open-source African stereotype dataset and evaluation framework grounded in local socio-cultural contexts. Through community engaged efforts across Senegal, Kenya, and Nigeria, we collected 1,163 stereotypes spanning gender, ethnicity, religion, age, and profession. Using few-shot prompting with human-in-the-loop validation, we augmented the dataset to over 5,000 stereotype-antistereotype pairs. Entries were validated through semantic clustering and manual annotation by culturally informed reviewers. Preliminary evaluation of language models reveals that nine of eleven models exhibit statistically significant bias, with Bias Preference Ratios (BPR) ranging from 0.63 to 0.78 (p <= 0.05), indicating systematic preferences for stereotypes over antistereotypes, particularly across age, profession, and gender dimensions. Domain-specific models appeared to show weaker bias in our setup, suggesting task-specific training may mitigate some associations. Looking ahead, AfriStereo opens pathways for future research on culturally grounded bias evaluation and mitigation, offering key methodologies for the AI community on building more equitable, context-aware, and globally inclusive NLP technologies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
