FastSpell:语言识别的魔法咒语
FastSpell: the LangId Magic Spell
AI总结:
针对现有语言识别工具难区分相似语言的问题,本文提出结合fastText和Hunspell的FastSpell工具,经基准测试验证其可提升相似语言识别效果,还能识别其他工具忽略的新语言。
摘要:
语言识别是语言资源自动化生产中的关键组成部分,尤其在多语言和大数据场景下更是如此。然而,常用的语言识别工具难以区分相似或关系密切的语言。本文提出了FastSpell,这是一种结合了fastText(一种预训练语言识别工具)和Hunspell(一种拼写检查器)的语言识别工具,其目的是在确定文本所属语言前提供经过优化的二次判断。我们描述了FastSpell算法,并解释了其使用和配置方法。为此,我们阐述了开发此类工具的必要性,并展示了一个基准测试,其中包含在FastSpell开发过程中评估的若干主流语言识别工具。我们证明,FastSpell不仅有助于提升相似语言的识别效果,还能识别其他工具忽略的新语言。
英文摘要:
Language identification is a crucial component in the automated production of language resources, particularly in multilingual and big data contexts. However, commonly used language identifiers struggle to differentiate between similar or closely-related languages. This paper introduces FastSpell, a language identifier that combines fastText (a pre-trained language identifier tool) and Hunspell (a spell checker) with the aim of having a refined second-opinion before deciding which language should be assigned to a text. We provide a description of the FastSpell algorithm along with an explanation on how to use and configure it. To that end, we motivate the need of such a tool and present a benchmark including some popular language identifiers evaluated during the development of FastSpell. We show how FastSpell is useful not only to improve identification of similar languages, but also to identify new ones ignored by other tools.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。