arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.07540cs.CLcs.LG
收藏

阿拉伯语文本分类中各种标记器的评价

Evaluating Various Tokenizers for Arabic Text Classification

King Fahd University of Petroleum and Minerals(法赫德国王石油与矿产大学)

Zaid Alyafeai, Maged S. Al-shaibani, Mustafa Ghaleb, Irfan Ahmad

摘要:

任何NLP管道中的第一步都是将文本拆分为单独的令牌。最明显和直接的方法是使用单词作为标记。然而,给定一个大的文本语料库,表示所有的单词在词汇量方面是没有效率的。在文献中,已经出现了许多标记化算法来解决这个问题,通过创建子词,这反过来又限制了给定文本语料库中的词汇量。大多数标记化技术是语言不可知的,即它们不包含给定语言的语言特征。更不用说在实践中评估这种技术的困难了。在本文中,我们介绍了三种新的阿拉伯语标记化算法,并将它们与其他三种使用无监督评估的基线进行比较。除此之外,我们比较了所有的六种算法,通过评估他们的三个监督分类任务,这是情感分析,新闻分类和诗歌分类使用六个公开可用的数据集。我们的实验表明,没有一种标记化技术是整体上的最佳选择,并且给定标记化算法的性能取决于数据集的大小,任务的类型以及数据集中存在的形态数量。然而,在各种文本分类任务中,一些标记化技术总体上比其他技术更好。

英文摘要:

The first step in any NLP pipeline is to split the text into individual tokens. The most obvious and straightforward approach is to use words as tokens. However, given a large text corpus, representing all the words is not efficient in terms of vocabulary size. In the literature, many tokenization algorithms have emerged to tackle this problem by creating subwords which in turn limits the vocabulary size in a given text corpus. Most tokenization techniques are language-agnostic i.e they don't incorporate the linguistic features of a given language. Not to mention the difficulty of evaluating such techniques in practice. In this paper, we introduce three new tokenization algorithms for Arabic and compare them to three other baselines using unsupervised evaluations. In addition to that, we compare all the six algorithms by evaluating them on three supervised classification tasks which are sentiment analysis, news classification and poetry classification using six publicly available datasets. Our experiments show that none of the tokenization technique is the best choice overall and that the performance of a given tokenization algorithm depends on the size of the dataset, type of the task, and the amount of morphology that exists in the dataset. However, some tokenization techniques are better overall as compared to others on various text classification tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑