Tokka-Bench：跨100种自然语言和20种编程语言的Tokenizer评估基准
Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages
浏览论文内容
中文总结 AI 辅助
本文提出Tokka-Bench，一个跨100种自然语言和20种编程语言的多指标分词器评估框架，比较七种BPE分词器，发现词汇分配策略比词汇量更关键，且编程语言效率已趋同。
中文摘要 AI 辅助
大型语言模型依赖子词分词器，其质量在不同语言间存在差异，但目前尚无标准化的多指标框架用于广泛的比较评估。我们推出了Tokka-Bench，一个开源框架，通过五个互补指标——每词元字节数、唯一词元覆盖率、子词繁殖率、词切分率和词汇构成——对分词器进行评估，覆盖100种自然语言（30多种文字系统）和20种编程语言，采用适应每种书写系统的语言感知切分方法。在单语言内比较七种BPE分词器（GPT-2、GPT-4、gpt-oss、Llama 3.1、Gemma 3、Qwen3和Kimi K2）时，我们发现词汇分配策略比原始词汇量大小更重要，且尽管自然语言特征存在差异，近期分词器在编程语言效率上已趋于一致。该框架、数据和交互式仪表板均已公开提供。
英文摘要
Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comparative evaluation. We introduce Tokka-Bench, an open-source framework that evaluates tokenizers on five complementary metrics -- bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary composition -- across 100 natural languages (30+ scripts) and 20 programming languages, using language-aware segmentation adapted to each writing system. Comparing seven BPE tokenizers (GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2) within individual languages, we find that vocabulary allocation strategy matters more than raw vocabulary size, and that programming-language efficiency has converged among recent tokenizers despite divergent natural-language profiles. The framework, data, and interactive dashboard are publicly available.
