arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.14134cs.CL
收藏

使用Punkt进行非拉丁语句子分割:对库尔德语(Sorani)文本的实验

Using Punkt for Sentence Segmentation in non-Latin Scripts: Experiments on Kurdish (Sorani) Texts

University of Kurdistan Hewlêr(埃尔比勒库尔德斯坦大学)

Roshna Omer Abdulrahman, Hossein Hassani

摘要:

切分是大多数自然语言处理任务的基本步骤。库尔德语是一种多方言,资源不足的语言,以不同的文字书写。缺乏各种切分语料库是库尔德语处理的主要瓶颈之一。我们使用Punkt,一种无监督的机器学习方法,对用波斯语-阿拉伯语书写的索拉尼方言的库尔德语语料库进行分割。根据文献,关于在非拉丁语数据上使用Punkt的研究很少。在我们的实验中,我们达到了91.10%的F1分数,出错率为16.32%。高出错率主要是由于库尔德语中的缩写情况,部分原因是序数词。这些数据可在https://github.com/KurdishBLARK/上公开获取,并根据CC BY-NC-SA 4.0许可证用于非商业用途。

英文摘要:

Segmentation is a fundamental step for most Natural Language Processing tasks. The Kurdish language is a multi-dialect, under-resourced language which is written in different scripts. The lack of various segmented corpora is one of the major bottlenecks in Kurdish language processing. We used Punkt, an unsupervised machine learning method, to segment a Kurdish corpus of Sorani dialect, written in Persian-Arabic script. According to the literature, studies on using Punkt on non-Latin data are scanty. In our experiment, we achieved an F1 score of 91.10% and had an Error Rate of 16.32%. The high Error Rate is mainly due to the situation of abbreviations in Kurdish and partly because of ordinal numerals. The data is publicly available at https://github.com/KurdishBLARK/ KTC-Segmented for non-commercial use under the CC BY-NC-SA 4.0 licence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑