Greed is All You Need：分词器推理方法的评估
Greed is All You Need: An Evaluation of Tokenizer Inference Methods
AI总结：
本文对七种分词器推理方法在四种算法和三种词表规模下进行受控分析，发现贪心推理在常用分词器上表现优异，而 SaGe 在形态学对齐上最佳。
摘要：
尽管 BPE 和 WordPiece 等子词分词器通常用于为 NLP 模型构建词表，但从这些词表将文本解码为 token 序列的方法往往未被明确指定，或与其构建方式不匹配。我们对四种不同算法和三种词表规模下的七种分词器推理方法进行了受控分析，该分析在我们为英语精心整理的一个全新内在评估套件上进行，结合了源于形态学、认知和信息理论的度量。我们表明，对于最常用的分词器，贪心推理表现惊人地好；而近期引入的上下文感知分词器 SaGe 在形态学对齐上优于所有其他方法。
英文摘要：
While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures rooted in morphology, cognition, and information theory. We show that for the most commonly used tokenizers, greedy inference performs surprisingly well; and that SaGe, a recently-introduced contextually-informed tokenizer, outperforms all others on morphological alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
