使用监督机器学习和特征组合的乌尔都语辱骂和威胁语言检测
Abusive and Threatening Language Detection in Urdu using Supervised Machine Learning and Feature Combinations
摘要：
本文介绍了在FIRE共享任务2021上提交的关于乌尔都语滥用和威胁性语言检测任务的系统描述。这项挑战旨在自动识别用乌尔都语撰写的辱骂和威胁性推文。我们提交的结果被选为第三次认可的竞争。本文报告了一个非详尽的实验列表，使我们能够达到提交的结果。此外，在比赛结果宣布后，我们设法取得了比提交的结果更好的结果。我们的模型在任务A（乌尔都语推文的辱骂性语言检测）和任务B（乌尔都语推文的威胁性语言检测）上分别获得了0.8318和0.4931的F1分数。结果表明，去除停用词，应用词形化，并通过n= 1，2，3的单词n-gram的组合创建特征向量的支持向量机对任务A产生了最好的结果。对于任务B，去除停用词的支持向量机，不应用词形化，从预训练的乌尔都语Word 2 Vec（关于单词一元语法和二元语法）创建特征向量，并使用过采样技术使数据集平衡，产生了最佳结果。代码可用于再现性。
英文摘要：
This paper presents the system descriptions submitted at the FIRE Shared Task 2021 on Urdu's Abusive and Threatening Language Detection Task. This challenge aims at automatically identifying abusive and threatening tweets written in Urdu. Our submitted results were selected for the third recognition at the competition. This paper reports a non-exhaustive list of experiments that allowed us to reach the submitted results. Moreover, after the result declaration of the competition, we managed to attain even better results than the submitted results. Our models achieved 0.8318 F1 score on Task A (Abusive Language Detection for Urdu Tweets) and 0.4931 F1 score on Task B (Threatening Language Detection for Urdu Tweets). Results show that Support Vector Machines with stopwords removed, lemmatization applied, and features vector created by the combinations of word n-grams for n=1,2,3 produced the best results for Task A. For Task B, Support Vector Machines with stopwords removed, lemmatization not applied, feature vector created from a pre-trained Urdu Word2Vec (on word unigrams and bigrams), and making the dataset balanced using oversampling technique produced the best results. The code is made available for reproducibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
