arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1701.03338cs.CL
收藏

LanideNN:字符窗口上的多语言识别

LanideNN: Multilingual Language Identification on Character Window

Charles University, Faculty of Mathematics and Physics(查理大学数学与物理学院);Institute of Formal and Applied Linguistics(形式与应用语言学研究所)

Tom Kocmi, Ondřej Bojar

摘要:

语言识别是自然语言处理的第一步,我们希望自动确定输入文本的语言。单语语言识别假设给定的文档是用一种语言编写的。在多语言识别中,文档通常是两种或三种语言,我们只需要它们的名称。我们的目标更进一步,并提出了一种文本语言识别的方法,其中语言可以任意改变,目标是识别每种语言的跨度。我们的方法是基于双向递归神经网络,它在单语和多语种的语言识别任务上表现良好,在6个数据集,涵盖131种语言。该方法对于短文档和跨域也保持了准确性,因此它是不准备训练数据的现成使用的理想方法。

英文摘要:

In language identification, a common first step in natural language processing, we want to automatically determine the language of some input text. Monolingual language identification assumes that the given document is written in one language. In multilingual language identification, the document is usually in two or three languages and we just want their names. We aim one step further and propose a method for textual language identification where languages can change arbitrarily and the goal is to identify the spans of each of the languages. Our method is based on Bidirectional Recurrent Neural Networks and it performs well in monolingual and multilingual language identification tasks on six datasets covering 131 languages. The method keeps the accuracy also for short documents and across domains, so it is ideal for off-the-shelf use without preparation of training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑