字符串有多长?对分词器的简要实证分析
How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers
AI总结:
本文通过实证分析揭示了不同分词器和文本分布下token长度的显著差异,挑战了传统对token长度的简化假设。
摘要:
前沿大语言模型在学术界、社会和工业中被日益广泛使用。一个常用的单位用于比较模型、它们的输入和输出以及估算推理定价的是token。通常,token被用作一种稳定的货币,假设其在不同的分词器和上下文中广泛一致,从而允许直接比较。然而,分词在模型和文本领域之间变化显著,使token计数的简单解释变得有问题。我们通过提供对分词的全面实证分析来量化这种变化,探索在不同文本数据分布下将序列压缩为token的过程。我们的分析挑战了关于token长度的常见启发式方法,发现它们过于简单化。我们希望本研究的见解能为当今大语言模型的分词提供更清晰和直观的理解。
英文摘要:
Frontier LLMs are increasingly utilised across academia, society and industry. A commonly used unit for comparing models, their inputs and outputs, and estimating inference pricing is the token. In general, tokens are used as a stable currency, assumed to be broadly consistent across tokenizers and contexts, enabling direct comparisons. However, tokenization varies significantly across models and domains of text, making naive interpretation of token counts problematic. We quantify this variation by providing a comprehensive empirical analysis of tokenization, exploring the compression of sequences to tokens across different distributions of textual data. Our analysis challenges commonly held heuristics about token lengths, finding them to be overly simplistic. We hope the insights of our study add clarity and intuition toward tokenization in contemporary LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。