发表机构
University of Illinois Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究揭示大型语言模型对非威望的上埃及阿拉伯语方言存在显著偏见，该偏见导致模型在句法评估和MMLU基准上的性能下降，凸显了亚方言变异对语言技术影响的重要性。
AI 中文摘要
先前关于埃及阿拉伯语的自然语言处理研究主要集中于具有威望地位的开罗埃及阿拉伯语（CEA）方言，导致在大型语言模型（LLM）和资源开发中缺乏对威望较低的上埃及（Sa'idi）埃及阿拉伯语（SEA）方言的表征。这种表征的缺失是否会影响 LLM 对 SEA 可接受性的看法（上游），以及针对 SEA 的上游偏见是否会导致较差的性能（下游）？我们在定向句法评估（TSE）任务中调查了 SEA 方言特征对 LLM 偏好的上游影响，该任务揭示了多个 LLM 对 SEA 存在显著偏见。随后，我们分析了这些相同特征对 MMLU 基准上模型下游性能的影响，并表明当模型面对 SEA 时，其性能会出现下降。这项工作强调了进一步探索亚方言变异如何影响语言技术的必要性。
英文摘要
Previous work on Egyptian Arabic in NLP has focused largely on the prestigious Cairene Egyptian Arabic (CEA) dialect, resulting in a lack of representation for the less prestigious Sa'idi Egyptian Arabic (SEA) dialect both in LLM and resource development. Does this lack of representation influence an LLM's view of the acceptability of SEA (upstream), and does an upstream bias against SEA lead to worse performance (downstream)? We investigate the upstream effect of SEA dialectal features on LLM preferences in a Targeted Syntactic Evaluation (TSE) task which reveals a significant bias against SEA across multiple LLMs. We then analyze the effect of these same features on downstream model performance on MMLU benchmarks and show that models experience a degradation in performance when presented with SEA. This work highlights the need for further exploration on how sub-dialectal variation impacts language technologies.
