方言毒性检测：评估跨语言变体的LLM-as-a-Judge一致性
Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties
AI总结：
本文评估了LLM-as-a-Judge在跨10个语言簇和60种变体多方言数据集上的毒性检测一致性，发现其最弱环节为LLM与人类的一致性，其次为方言一致性。
摘要：
关于方言差异如何影响现代LLM毒性检测，目前缺乏系统研究。此外，尽管使用LLM作为评估者（“LLM-as-a-judge”）是一个不断发展的研究领域，但它们对方言细微差别的敏感性仍探讨不足，需要更多关注。本文通过对LLM跨多种方言的全面毒性评估填补了这些空白。我们通过合成转换和人工辅助翻译创建了一个多方言数据集，涵盖10个语言簇和60种变体。随后，我们评估了三个LLM在多语言、方言以及LLM与人类一致性方面评估毒性的能力。研究结果表明，LLM在处理多语言和方言变体时具有敏感性。然而，若必须对一致性进行排名，最弱的领域是LLM与人类的共识，其次是方言一致性。代码仓库：\url{https://github.com/ffaisal93/dialect_toxicity_llm_judge}
英文摘要：
There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. Furthermore, although using LLMs as evaluators ("LLM-as-a-judge") is a growing research area, their sensitivity to dialectal nuances is still underexplored and requires more focused attention. In this paper, we address these gaps through a comprehensive toxicity evaluation of LLMs across diverse dialects. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluated three LLMs on their ability to assess toxicity across multilingual, dialectal, and LLM-human consistency. Our findings show that LLMs are sensitive in handling both multilingual and dialectal variations. However, if we have to rank the consistency, the weakest area is LLM-human agreement, followed by dialectal consistency. Code repository: \url{https://github.com/ffaisal93/dialect_toxicity_llm_judge}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
