印度-雅利安语中的仇恨言论与冒犯性内容检测：LSTM 与 Transformers 的较量
Hate Speech and Offensive Content Detection in Indo-Aryan Languages: A Battle of LSTM and Transformers
AI总结：
本文面向 HASOC 2023，比较 BERT 变体、XLM-R 与 LSTM 在五种印度-雅利安语仇恨言论检测中的表现，发现最优模型因语言而异。
摘要：
社交媒体平台是个人表达想法和经历的便捷渠道，导致各年龄段用户生成数据大量涌入。这些平台虽支持自由表达，也带来重大挑战，包括仇恨言论和冒犯性内容的扩散。此类令人反感的语言会破坏客观讨论，并可能使辩论走向激进化，最终威胁民主价值观。因此，各组织已采取措施监控和遏制辱骂行为，这就需要用于识别可疑帖子的自动化方法。本文为英语和印度-雅利安语仇恨言论与冒犯性内容识别（HASOC）2023 共享任务赛道作出贡献。我们 Z-AGI Labs 团队针对五种不同语言——孟加拉语、阿萨姆语、博多语、僧伽罗语和古吉拉特语——开展了全面的仇恨言论分类比较分析。研究涵盖多种预训练模型，包括 BERT 变体、XLM-R 和 LSTM 模型，以评估它们在这些语言中识别仇恨言论的性能。结果揭示了模型性能方面有趣的差异。值得注意的是，Bert Base Multilingual Cased 在多种语言中表现强劲，在孟加拉语和阿萨姆语上分别取得 0.67027 和 0.70525 的 F1 分数；同时，它在博多语上以 0.83009 的可观 F1 分数显著优于其他模型。在僧伽罗语中，XLM-R 表现突出，F1 分数为 0.83493；而在古吉拉特语中，一个基于 LSTM 的定制模型表现最佳，F1 分数为 0.76601。本研究为各种预训练模型在多语言环境下用于仇恨言论检测的适用性提供了有价值的洞见。通过考虑每种语言的细微差别，我们的研究有助于在构建稳健的仇恨言论检测系统时进行有依据的模型选择。
英文摘要：
Social media platforms serve as accessible outlets for individuals to express their thoughts and experiences, resulting in an influx of user-generated data spanning all age groups. While these platforms enable free expression, they also present significant challenges, including the proliferation of hate speech and offensive content. Such objectionable language disrupts objective discourse and can lead to radicalization of debates, ultimately threatening democratic values. Consequently, organizations have taken steps to monitor and curb abusive behavior, necessitating automated methods for identifying suspicious posts. This paper contributes to Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages (HASOC) 2023 shared tasks track. We, team Z-AGI Labs, conduct a comprehensive comparative analysis of hate speech classification across five distinct languages: Bengali, Assamese, Bodo, Sinhala, and Gujarati. Our study encompasses a wide range of pre-trained models, including Bert variants, XLM-R, and LSTM models, to assess their performance in identifying hate speech across these languages. Results reveal intriguing variations in model performance. Notably, Bert Base Multilingual Cased emerges as a strong performer across languages, achieving an F1 score of 0.67027 for Bengali and 0.70525 for Assamese. At the same time, it significantly outperforms other models with an impressive F1 score of 0.83009 for Bodo. In Sinhala, XLM-R stands out with an F1 score of 0.83493, whereas for Gujarati, a custom LSTM-based model outshined with an F1 score of 0.76601. This study offers valuable insights into the suitability of various pre-trained models for hate speech detection in multilingual settings. By considering the nuances of each, our research contributes to an informed model selection for building robust hate speech detection systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
