arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.01322cs.CL
收藏

PALI:一个针对波斯-阿拉伯字母的语言识别基准

PALI: A Language Identification Benchmark for Perso-Arabic Scripts

George Mason University(乔治梅森大学)

Sina Ahmadi, Milind Agarwal, Antonios Anastasopoulos

AI总结:

本文针对波斯-阿拉伯字母的语言识别挑战,提出使用监督技术进行句子分类,并构建了处理易混淆语言簇的分层模型,实验证明了其有效性。

摘要:

波斯-阿拉伯字母是全球各语言社区广泛采用并使用的一类字母。使用此类字母识别各种语言对语言技术至关重要,且在低资源设置下具有挑战性。因此,本文阐明了使用波斯-阿拉伯字母检测语言的挑战,特别是在存在“非传统”书写方式的双语社区中。为了解决这一问题,我们使用一组监督技术将句子分类为对应的语言。在此基础上,我们还提出了一种分层模型,针对经常被分类器混淆的语言簇进行专门处理。我们的实验结果表明了所提解决方案的有效性。

英文摘要:

The Perso-Arabic scripts are a family of scripts that are widely adopted and used by various linguistic communities around the globe. Identifying various languages using such scripts is crucial to language technologies and challenging in low-resource setups. As such, this paper sheds light on the challenges of detecting languages using Perso-Arabic scripts, especially in bilingual communities where ``unconventional'' writing is practiced. To address this, we use a set of supervised techniques to classify sentences into their languages. Building on these, we also propose a hierarchical model that targets clusters of languages that are more often confused by the classifiers. Our experiment results indicate the effectiveness of our solutions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑