地理信息感知的语言识别
Geographically-Informed Language Identification
AI总结：
本文提出一种地理信息感知的语言识别方法，通过构建16个区域模型并纳入国际语言，在传统测试数据上F1提升1.7至10.4个百分点，覆盖916种语言，显著改善真实语料库的语言标签。
摘要：
本文开发了一种语言识别方法，其中模型所考虑的语言集合取决于文本的地理来源。鉴于许多数字语料库可以在国家层面进行地理定位，本文构建了16个区域特定模型，每个模型包含预期在该区域内国家出现的语言。这些区域模型还各自包含31种广泛使用的国际语言，以确保无论位置如何都能覆盖这些通用语。使用传统语言识别测试数据的上游评估显示，F1分数提高了1.7个百分点（东南亚）至10.4个百分点（北非）。对社交媒体数据的下游评估表明，这种改进的性能对应用于大型真实世界语料库的语言标签产生了显著影响。结果是一个高精度的模型，在50个字符的样本量下覆盖916种语言，通过将地理信息纳入模型，性能得到了提升。
英文摘要：
This paper develops an approach to language identification in which the set of languages considered by the model depends on the geographic origin of the text in question. Given that many digital corpora can be geo-referenced at the country level, this paper formulates 16 region-specific models, each of which contains the languages expected to appear in countries within that region. These regional models also each include 31 widely-spoken international languages in order to ensure coverage of these linguae francae regardless of location. An upstream evaluation using traditional language identification testing data shows an improvement in f-score ranging from 1.7 points (Southeast Asia) to as much as 10.4 points (North Africa). A downstream evaluation on social media data shows that this improved performance has a significant impact on the language labels which are applied to large real-world corpora. The result is a highly-accurate model that covers 916 languages at a sample size of 50 characters, the performance improved by incorporating geographic information into the model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
