用于单词相似度预测的特定于单词的标记化
Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction
摘要：
不需要标记语言资源或词典的独立标记化（LIT）方法最近因其在资源贫乏语言中的适用性而受到欢迎。此外，它们使用固定大小的词汇表来表示一种语言，并且可以有效地处理看不见的或罕见的单词。另一方面，语言特定的标记化（LST）方法有着悠久的历史，并且是使用精心创建的词汇和培训资源开发的。与LIT方法产生的子标记不同，LST方法产生有效的形态子字。尽管LIT与LST方法之间存在对比，但它们在下游NLP任务上的性能仍然不清楚。在本文中，我们经验比较这两种方法使用语义相似性测量作为评估任务，在不同的语言。我们的实验结果涵盖八种语言表明，LST一贯优于LIT时，词汇量大，但LIT可以产生可比或更好的结果比LST在许多语言相对较小（即小于10万字）的词汇量，鼓励使用LIT时，语言特定的资源不可用，不完整或需要一个较小的模型。此外，我们发现，平滑逆频率（SIF）是一个准确的方法来创建词嵌入从子词嵌入的多语言语义相似度预测任务。进一步分析的最近邻的令牌表明，语义和句法相关的令牌紧密嵌入子词嵌入空间
英文摘要：
Language-independent tokenisation (LIT) methods that do not require labelled language resources or lexicons have recently gained popularity because of their applicability in resource-poor languages. Moreover, they compactly represent a language using a fixed size vocabulary and can efficiently handle unseen or rare words. On the other hand, language-specific tokenisation (LST) methods have a long and established history, and are developed using carefully created lexicons and training resources. Unlike subtokens produced by LIT methods, LST methods produce valid morphological subwords. Despite the contrasting trade-offs between LIT vs. LST methods, their performance on downstream NLP tasks remain unclear. In this paper, we empirically compare the two approaches using semantic similarity measurement as an evaluation task across a diverse set of languages. Our experimental results covering eight languages show that LST consistently outperforms LIT when the vocabulary size is large, but LIT can produce comparable or better results than LST in many languages with comparatively smaller (i.e. less than 100K words) vocabulary sizes, encouraging the use of LIT when language-specific resources are unavailable, incomplete or a smaller model is required. Moreover, we find that smoothed inverse frequency (SIF) to be an accurate method to create word embeddings from subword embeddings for multilingual semantic similarity prediction tasks. Further analysis of the nearest neighbours of tokens show that semantically and syntactically related tokens are closely embedded in subword embedding spaces
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
