arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16241cs.CLcs.AI
收藏

大型语言模型是否准备好取代孟加拉语标注者?

Are LLMs Ready to Replace Bangla Annotators?

Md. Najib Hasan, Touseef Hasan, Souvika Sarkar

AI总结:

本研究探讨了大型语言模型在孟加拉语仇恨言论标注任务中的表现,发现模型规模并非决定标注质量的唯一因素,且存在显著的标注偏差和不稳定性。

摘要:

大型语言模型(LLMs)越来越多地被用作自动化标注工具来扩大数据集的创建规模,但它们作为无偏标注工具的可靠性——尤其是在低资源和身份敏感的环境中——仍知之甚少。在本工作中,我们研究了LLMs作为零样本标注工具在孟加拉语仇恨言论任务中的行为,这是一个即使对于人类来说都具有挑战性的任务,标注者偏见可能产生严重后续影响。我们使用统一的评估框架对17个LLMs进行了系统基准测试。我们的分析揭示了标注者偏见和模型判断中的显著不稳定性。令人惊讶的是,模型规模的增加并不保证标注质量的提高——较小、更符合任务的模型经常表现出更一致的行为。这些结果突显了当前LLMs在低资源语言敏感标注任务中的重要限制,并强调了在部署前进行仔细评估的必要性。

英文摘要:

Large Language Models (LLMs) are increasingly used as automated annotators to scale dataset creation, yet their reliability as unbiased annotators--especially for low-resource and identity-sensitive settings--remains poorly understood. In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences. We conduct a systematic benchmark of 17 LLMs using a unified evaluation framework. Our analysis uncovers annotator bias and substantial instability in model judgments. Surprisingly, increased model scale does not guarantee improved annotation quality--smaller, more task-aligned models frequently exhibit more consistent behavior than their larger counterparts. These results highlight important limitations of current LLMs for sensitive annotation tasks in low-resource languages and underscore the need for careful evaluation before deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑