arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.14263cs.CLcs.AI
收藏

LIMIT:基于分层模型的350+种语言的语言识别、误识别与翻译

LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages

George Mason University(乔治梅森大学)

Milind Agarwal, Md Mahfuz Ibn Alam, Antonios Anastasopoulos

AI总结:

针对现有语言识别系统难以覆盖多数低资源语言的问题,研究构建了含350+语言的MCS-350语料库,提出误预测解决分层模型LIMIt,显著降低识别错误率,无需重训大模型即可扩展低资源语言覆盖。

摘要:

了解输入文本/音频的语言,是使用几乎所有自然语言处理(NLP)工具(如标注器、解析器或翻译系统)的必要第一步。语言识别是一个已被充分研究的问题,有时甚至被认为已经解决;但实际上,由于数据匮乏和计算挑战,当前系统无法准确识别全球7000种语言中的大多数。为解决这一瓶颈,我们首先构建了一个语料库MCS-350,包含350多种语言的5万篇多语言平行儿童故事。MCS-350可作为短文本语言识别以及低资源印度语言和非洲语言的1400多个新翻译方向的基准。其次,我们提出了一种新型的误预测解决分层模型LIMIt用于语言识别,该模型在我们构建的儿童故事数据集上将错误率降低了55%(从0.71降至0.32),在FLORES-200基准上将错误率降低了40%(从0.23降至0.14)。我们的方法仅依靠系统性误预测模式,无需从头重新训练大型模型,就能将语言识别的覆盖范围扩展到低资源语言。

英文摘要:

Knowing the language of an input text/audio is a necessary first step for using almost every NLP tool such as taggers, parsers, or translation systems. Language identification is a well-studied problem, sometimes even considered solved; in reality, due to lack of data and computational challenges, current systems cannot accurately identify most of the world's 7000 languages. To tackle this bottleneck, we first compile a corpus, MCS-350, of 50K multilingual and parallel children's stories in 350+ languages. MCS-350 can serve as a benchmark for language identification of short texts and for 1400+ new translation directions in low-resource Indian and African languages. Second, we propose a novel misprediction-resolution hierarchical model, LIMIt, for language identification that reduces error by 55% (from 0.71 to 0.32) on our compiled children's stories dataset and by 40% (from 0.23 to 0.14) on the FLORES-200 benchmark. Our method can expand language identification coverage into low-resource languages by relying solely on systemic misprediction patterns, bypassing the need to retrain large models from scratch.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑