arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1707.04817cs.CL
收藏

开集语言识别

Open-Set Language Identification

Harvard Medical School(哈佛医学院)

Shervin Malmasi

摘要:

我们提出了第一个开放集的语言识别实验,使用一类分类。我们首先强调传统的特征提取方法的缺点,并提出了一种基于哈希的特征向量化方法作为解决方案。使用来自不同书写系统的10种语言的数据集,我们仅使用每种语言的单语语料库来训练一类支持向量机。每个模型都针对来自所有10种语言的测试数据集进行评估,我们的平均F分数为0.99,突出了这种方法在开集语言识别中的有效性。

英文摘要:

We present the first open-set language identification experiments using one-class classification. We first highlight the shortcomings of traditional feature extraction methods and propose a hashing-based feature vectorization approach as a solution. Using a dataset of 10 languages from different writing systems, we train a One- Class Support Vector Machine using only a monolingual corpus for each language. Each model is evaluated against a test set of data from all 10 languages and we achieve an average F-score of 0.99, highlighting the effectiveness of this approach for open-set language identification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑