一种快速、紧凑、准确的混码文本语言识别模型
A Fast, Compact, Accurate Model for Language Identification of Codemixed Text
摘要:
我们解决了细粒度的多语言识别:为句子中的每个标记提供语言代码,包括包含多种语言的代码混合文本。这种文本在网上、文档、社交媒体和留言板中很流行。我们证明了一个简单的全局约束解码器的前馈网络可以准确,快速地标记100种语言和100种语言对的混合和单语文本。该模型在准确性和速度方面优于先前发布的多语言方法,在三个代码混合数据集上获得了800倍的速度和19.5%的平均绝对增益。此外,它在单语语言识别方面优于几个基准系统。
英文摘要:
We address fine-grained multilingual language identification: providing a language code for every token in a sentence, including codemixed text containing multiple languages. Such text is prevalent online, in documents, social media, and message boards. We show that a feed-forward network with a simple globally constrained decoder can accurately and rapidly label both codemixed and monolingual text in 100 languages and 100 language pairs. This model outperforms previously published multilingual approaches in terms of both accuracy and speed, yielding an 800x speed-up and a 19.5% averaged absolute gain on three codemixed datasets. It furthermore outperforms several benchmark systems on monolingual language identification.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。