arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2508.11166cs.CL

克服图鲁语低资源障碍:用于冒犯性语言识别的神经模型与语料库构建

Overcoming Low-Resource Barriers in Tulu: Neural Models and Corpus Creation for OffensiveLanguage Identification

Anusha M D, Deepthi Vikram, Bharathi Raja Chakravarthi, Parameshwar R Hegde

AI总结:

本文构建了首个代码混合图鲁语冒犯性语言识别基准数据集,并实验证明带自注意力机制的BiGRU模型优于Transformer模型,为低资源语言NLP研究奠定基础。

摘要:

图鲁语是一种主要在印度南部使用的低资源达罗毗荼语系语言,尽管其数字存在感日益增强,但计算资源仍然有限。本研究提出了首个用于代码混合图鲁语社交媒体内容中冒犯性语言识别(OLI)的基准数据集,该数据集收集自多个领域的YouTube评论。该数据集具有较高的标注者间一致性(Krippendorff's alpha = 0.984),包含3,845条评论,分为四个类别:非冒犯性、非图鲁语、无特定目标的冒犯性和有特定目标的冒犯性。我们评估了一系列深度学习模型,包括GRU、LSTM、BiGRU、BiLSTM、CNN以及基于注意力机制的变体,同时评估了Transformer架构(mBERT、XLM-RoBERTa)。带有自注意力机制的BiGRU模型取得了最佳性能,准确率为82%,宏平均F1值为0.81。Transformer模型表现不佳,凸显了多语言预训练在代码混合、资源匮乏环境中的局限性。这项工作为进一步开展图鲁语及类似低资源、代码混合语言的NLP研究奠定了基础。

英文摘要:

Tulu, a low-resource Dravidian language predominantly spoken in southern India, has limited computational resources despite its growing digital presence. This study presents the first benchmark dataset for Offensive Language Identification (OLI) in code-mixed Tulu social media content, collected from YouTube comments across various domains. The dataset, annotated with high inter-annotator agreement (Krippendorff's alpha = 0.984), includes 3,845 comments categorized into four classes: Not Offensive, Not Tulu, Offensive Untargeted, and Offensive Targeted. We evaluate a suite of deep learning models, including GRU, LSTM, BiGRU, BiLSTM, CNN, and attention-based variants, alongside transformer architectures (mBERT, XLM-RoBERTa). The BiGRU model with self-attention achieves the best performance with 82% accuracy and a 0.81 macro F1-score. Transformer models underperform, highlighting the limitations of multilingual pretraining in code-mixed, under-resourced contexts. This work lays the foundation for further NLP research in Tulu and similar low-resource, code-mixed languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑