arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.00962cs.CL
收藏

面向知识与推理基准的大语言模型方言偏见分析

Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks

Department of Information Science, Cornell University(康奈尔大学信息科学系);Apple(苹果公司);Cornell Tech(康奈尔科技)

Eileen Pan, Anna Seo Gyeong Choi, Maartje ter Hoeve, Skyler Seto, Allison Koenecke

AI总结:

该研究分析了LLM在知识与推理基准任务中的方言偏见,发现非标准英语方言会使多项选择题准确率最高降20%,且三条特定语法规则是性能下降的主要原因,呼吁聚焦高影响语法结构研究偏见缓解方法。

摘要:

大语言模型(LLM)在现代自然语言处理中应用十分广泛。但已有研究表明,LLM在处理代表性不足的英语方言时性能会出现下降。我们分析了将“标准”美式英语问题改写为非“标准”方言变体对多项选择题问答任务的影响,发现准确率最多可下降20%。此外,我们还探究了非“标准”英语问题下性能不佳的语法层面成因。我们发现不同的语法规则对性能的影响各不相同,但部分规则的影响更为显著:三条特定语法规则(存在句“it”、零系词、y'all)可以解释多方言中观测到的大部分性能下降现象。我们呼吁未来研究聚焦于单个高影响语法结构,探索偏见缓解方法。

英文摘要:

Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-"standard" English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential "it", zero copula, and y'all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑