解析分词（Tokenization）：评估文本压缩率及其与模型性能的相关性
Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance
AI总结：
该研究通过控制训练数据量调整BPE分词器的压缩能力，结合多语言实验证实分词压缩率与预训练语言模型下游性能正相关，指出提升分词器压缩能力是优化模型性能的有效途径。
摘要：
尽管压缩是最常用的分词算法BPE（字节对编码）的核心基石，但压缩在分词过程中的重要性仍不明确。本文从理论层面论证了压缩的重要性：压缩可被视为0元语法语言建模，即给所有词元分配相等概率。我们还通过实证证明了压缩对预训练语言模型下游任务表现的重要性。我们通过改变训练BPE分词器时可用的文档数量来控制其压缩能力：文档数量从100万份到等同于无训练数据的基于字符的分词器不等。随后我们基于这些分词器预训练了英语语言模型，并在多个任务上进行微调。研究表明，分词器的压缩率与模型的下游性能存在相关性，说明压缩是衡量分词质量的可靠内在指标。这种相关性在生成任务（强于分类任务）或较小模型（强于大模型）中更为显著。我们在土耳其语上重复了部分代表性实验，得到了相似结果，证实该结论适用于类型学特征与英语不同的语言。我们总结认为，构建压缩能力更强的分词器是未来研究和提升模型整体性能的有效方向。
英文摘要：
Despite it being the cornerstone of BPE, the most common tokenization algorithm, the importance of compression in the tokenization process is still unclear. In this paper, we argue for the theoretical importance of compression, that can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. We also demonstrate the empirical importance of compression for downstream success of pre-trained language models. We control the compression ability of several BPE tokenizers by varying the amount of documents available during their training: from 1 million documents to a character-based tokenizer equivalent to no training data at all. We then pre-train English language models based on those tokenizers and fine-tune them over several tasks. We show that there is a correlation between tokenizers' compression and models' downstream performance, suggesting that compression is a reliable intrinsic indicator of tokenization quality. These correlations are more pronounced for generation tasks (over classification) or for smaller models (over large ones). We replicated a representative part of our experiments on Turkish and found similar results, confirming that our results hold for languages with typological characteristics dissimilar to English. We conclude that building better compressing tokenizers is a fruitful avenue for further research and for improving overall model performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
