arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.11744cs.CLcs.LG
收藏

AfroLID:非洲语言的神经语言识别工具

AfroLID: A Neural Language Identification Tool for African Languages

The University of British Columbia(不列颠哥伦比亚大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, Alcides Alcoba Inciarte

摘要:

语言识别(LID)是NLP的重要前提,特别是对于挖掘Web数据。问题是,当今世界上7000多种语言中的大多数都没有被LID技术覆盖。我们通过引入AfroLID来解决非洲的这一紧迫问题,AfroLID是一个针对$517$非洲语言和品种的神经LID工具包。AfroLID利用了一个多域网络数据集,该数据集使用五个正字法系统从14个语言家族中手动策划。当在我们的盲测集上进行评估时,AfroLID达到95.89 F_1分数。我们还将AfroLID与五种现有的LID工具进行了比较,每种工具都覆盖了少量的非洲语言,发现它在大多数语言上的表现都优于它们。我们进一步展示了AfroLID在野外的实用性,通过在服务严重不足的Twitter域上进行测试。最后,我们提供了一些受控的案例研究,并进行了语言动机的错误分析,使我们能够展示AfroLID的强大功能和局限性。

英文摘要:

Language identification (LID) is a crucial precursor for NLP, especially for mining web data. Problematically, most of the world's 7000+ languages today are not covered by LID technologies. We address this pressing issue for Africa by introducing AfroLID, a neural LID toolkit for $517$ African languages and varieties. AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems. When evaluated on our blind Test set, AfroLID achieves 95.89 F_1-score. We also compare AfroLID to five existing LID tools that each cover a small number of African languages, finding it to outperform them on most languages. We further show the utility of AfroLID in the wild by testing it on the acutely under-served Twitter domain. Finally, we offer a number of controlled case studies and perform a linguistically-motivated error analysis that allow us to both showcase AfroLID's powerful capabilities and limitations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑