印地语方言偏见评估：整合基于RAG的翻译和人类增强的RLAIF的多阶段框架
Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF
AI总结：
本文提出一个多阶段框架，通过基于RAG的翻译和人类增强的RLAIF评估九种印地语方言的偏见。研究发现方言差异导致性能下降，模型规模增加不总是缓解偏见。
摘要：
大型语言模型（LLMs）经常表现出对低资源语言区域方言的性能偏见。然而，量化这些差异的框架仍然稀缺。我们提出一个两阶段框架，评估LLM在九种印地语方言上的问答偏见。首先，我们通过检索增强生成（RAG）流程将标准印地语问题翻译并标注为方言变体，生成4000个问题集。由于传统翻译质量评估指标在非标准化方言上失效，我们使用LLM作为判断者来评估保真度，其与人类的相关性证实优于传统指标。其次，我们对19个LLM在这些标注集上进行基准测试，运行68,395次RLAIF评估，通过多判官一致性和人类回退验证。我们的发现揭示了严重性能下降与语言差异的联系。例如，对高度差异化的切蒂甘方言的响应得分为5.44/10，而塔尼加尔方言得分为7.68/10。此外，模型规模的增加并不总是缓解这种偏见。我们贡献了经过验证的翻译质量评估方法、严格基准数据集以及用于安全关键应用的临界偏见敏感度（CBS）度量标准。
英文摘要：
Large language models (LLMs) frequently exhibit performance biases against regional dialects of low-resource languages. However, frameworks to quantify these disparities remain scarce. We propose a two-phase framework to evaluate dialectal bias, operationalized as comprehension degradation relative to standard Bengali, in LLM question-answering across nine Bengali dialects. First, we translate and gold-label standard Bengali questions into dialectal variants adopting a retrieval-augmented generation (RAG) pipeline to prepare 4,000 question sets. Since traditional translation quality evaluation metrics fail on unstandardized dialects, we evaluate fidelity using an LLM-as-a-judge, which human correlation confirms outperforms legacy metrics. Second, we benchmark 19 LLMs across these gold-labeled sets, running 68,395 RLAIF evaluations validated through multi-judge agreement and human fallback. Our findings reveal severe performance drops linked to linguistic divergence. For instance, responses to the highly divergent Chittagong dialect score 5.44/10, compared to 7.68/10 for Tangail. Furthermore, increased model scale does not consistently mitigate this bias. We contribute a validated translation quality evaluation method, a rigorous benchmark dataset, and a Critical Bias Sensitivity (CBS) metric for safety-critical applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
