非裔美国人语言偏见在自然语言生成中的评估
Evaluation of African American Language Bias in Natural Language Generation
AI总结：
本研究评估大型语言模型在非裔美国人语言与白人主流英语间的理解差异，通过两个生成任务和六种模型，构建新数据集并揭示模型对AAL的偏见与理解不足。
摘要：
我们评估了大型语言模型（LLMs）对非裔美国人语言（AAL）的理解程度，并将其与白人主流英语（WME）——美国课堂中鼓励的“标准”英语形式——上的表现进行比较。我们使用自动指标和人工判断来衡量LLM在两个任务上的表现：对应生成任务（模型根据WME生成AAL，或根据AAL生成WME）和掩码跨度预测（MSP）任务（模型预测从输入中移除的短语）。我们的贡献包括：（1）评估六个预训练大型语言模型在两个语言生成任务上的表现；（2）构建了一个包含多种语境（社交媒体、嘻哈歌词、焦点小组和语言学访谈）的AAL文本新数据集，并附有对应的人工标注WME版本；（3）记录了模型性能差距，这些差距表明存在偏见，并识别了模型对AAL特征理解不足的趋势。
英文摘要：
We evaluate how well LLMs understand African American Language (AAL) in comparison to their performance on White Mainstream English (WME), the encouraged "standard" form of English taught in American classrooms. We measure LLM performance using automatic metrics and human judgments for two tasks: a counterpart generation task, where a model generates AAL (or WME) given WME (or AAL), and a masked span prediction (MSP) task, where models predict a phrase that was removed from their input. Our contributions include: (1) evaluation of six pre-trained, large language models on the two language generation tasks; (2) a novel dataset of AAL text from multiple contexts (social media, hip-hop lyrics, focus groups, and linguistic interviews) with human-annotated counterparts in WME; and (3) documentation of model performance gaps that suggest bias and identification of trends in lack of understanding of AAL features.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
