前沿大语言模型在阿拉伯文化与社会语言学知识上的基准测试：基于人类专家真实标注的交叉评估框架
Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth
AI总结：
提出一个交叉评估框架，通过人类专家标注的103个提示-评分标准对，评估前沿LLM在埃及和伊拉克阿拉伯语文化与社会语言学知识上的表现，发现GPT-5.4是最可靠的自动评判者，文化任务比语言任务更难评分，且模型在埃及提示上表现优于伊拉克提示。
摘要：
人类专家评估的成本是将语言模型部署到专业、高风险领域的主要瓶颈。对于阿拉伯社会语言学知识而言，这一点尤为突出：可信的评分不仅需要语言流利度，还需要无法通过表面指标近似获得的深层文化熟悉度。我们通过一个交叉评估框架来解决这一问题，该框架针对两个代表性不足的阿拉伯方言社区：埃及阿拉伯语和伊拉克阿拉伯语。我们贡献了103个经过验证的提示-评分标准对（70个埃及，33个伊拉克；53个文化，50个语言），由母语专家使用惩罚加权评分标准编写和评分，该评分标准区分了正面内容要求和答案特定的负面错误标准。三个前沿LLM作为目标模型（由人类专家对302个独特的提示-响应对进行评分），而五个前沿LLM作为自动评判者，执行提供者级别的自我评估防护。一种结合平均绝对偏差（MAD）和符号平均误差的双指标方案将方向性评分偏差与对称噪声分开。在1,307个评判者评估中：GPT-5.4是最可靠的评判者（MADj = 10.21个百分点，符号误差 = -1.12%）；五个评判者中有四个表现出系统性宽容（+2.01%至+6.56%）；所有评判者对文化任务的评分难度均高于语言任务（MAD差距1.83-4.78个百分点）；模型在埃及提示上的表现显著优于伊拉克提示。然而，考虑到伊拉克和埃及专家之间的宽容度差异，我们不能将这一差距完全归因于模型知识。因此，我们强调那些不假设人类评分者之间具有相同宽容度的发现。在所有样本中，隐含的文化推理——要求模型模拟母语者的判断而非依赖词汇验证——成为所有评判者模型在自动评分中的主要失败模式。
英文摘要：
The cost of human expert evaluation is a principal bottleneck to deploying language models in specialized, high-stakes domains. This is particularly acute for Arabic sociolinguistic knowledge: credible grading requires not only linguistic fluency but deep cultural familiarity that cannot be approximated by surface-level metrics. We address this with a cross-evaluation framework instantiated on two underrepresented Arabic dialect communities: Egyptian and Iraqi Arabic. We contribute 103 validated prompt-rubric pairs (70 Egyptian, 33 Iraqi; 53 Cultural, 50 Linguistic), authored and graded by native-speaker SMEs using penalty-weighted rubrics distinguishing positive content requirements from answer-specific negative error criteria. Three frontier LLMs serve as target models (graded by human SMEs across 302 unique prompt-response pairs), while five frontier LLMs serve as automated judges enforcing a provider-level self-evaluation guard. A dual-metric scheme combining Mean Absolute Deviation (MAD) with Signed Mean Error separates directional grading bias from symmetric noise. Across 1,307 judge evaluations: GPT-5.4 is the most reliable judge (MADj = 10.21 pp, Signed Error = -1.12%); four of five judges show systematic leniency (+2.01% to +6.56%); Cultural tasks are harder to grade than Linguistic tasks for all judges (MAD gap 1.83-4.78 pp); and models substantially outperform on Egyptian prompts compared to Iraqi prompts. However, given leniency differences between Iraqi and Egyptian SMEs, we cannot solely attribute this gap to model knowledge. We therefore emphasize findings that do not assume identical leniency across human graders. Across all samples, implicit cultural reasoning -- requiring models to simulate native-speaker judgment rather than rely on lexical verification -- emerges as the primary failure mode for automated grading across all judge models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
