通过空间的替代处理来改进令牌化
Improving Tokenisation by Alternative Treatment of Spaces
摘要：
标记化是几乎所有NLP任务的第一步，最先进的基于transformer的语言模型都使用子词标记化算法来处理输入文本。现有的算法存在问题，通常会产生语言有效性有限的标记，并根据它们在单词中的位置不同来表示等效字符串。我们假设这些问题阻碍了基于transformer的模型处理复杂单词的能力，并建议这些问题是允许令牌包含空格的结果。因此，我们尝试了另一种标记化方法，其中空格始终被视为单独的标记。具体来说，我们将此修改应用于BPE和Unigram算法。我们发现，我们的修改后的算法导致下游NLP任务，涉及处理复杂的单词，同时在一般的自然语言理解任务的性能没有不利影响的性能提高。本质上，我们发现我们修改后的算法提供了更多的形态正确的标记，特别是在处理前缀。鉴于我们的实验结果，我们主张始终将空间视为单独的令牌，作为改进的令牌化方法。
英文摘要：
Tokenisation is the first step in almost all NLP tasks, and state-of-the-art transformer-based language models all use subword tokenisation algorithms to process input text. Existing algorithms have problems, often producing tokenisations of limited linguistic validity, and representing equivalent strings differently depending on their position within a word. We hypothesise that these problems hinder the ability of transformer-based models to handle complex words, and suggest that these problems are a result of allowing tokens to include spaces. We thus experiment with an alternative tokenisation approach where spaces are always treated as individual tokens. Specifically, we apply this modification to the BPE and Unigram algorithms. We find that our modified algorithms lead to improved performance on downstream NLP tasks that involve handling complex words, whilst having no detrimental effect on performance in general natural language understanding tasks. Intrinsically, we find our modified algorithms give more morphologically correct tokenisations, in particular when handling prefixes. Given the results of our experiments, we advocate for always treating spaces as individual tokens as an improved tokenisation method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
