打破词元天花板：蒸馏更小更强的字节模型
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
- University of Washington（华盛顿大学）
- Meta FAIR
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文提出将字节模型蒸馏为更小更强模型的方法，通过高效转换词元对数几率为字节对数几率，证明字节模型在更高计算量下性能超越词元模型，且数据效率更高，存储成本更低。
AI中文摘要：
小型模型通过从共享其词元化方案的较大模型进行蒸馏而变得更具能力。然而，随着计算量和数据量的增加，蒸馏后的字节模型和词元模型在扩展趋势上的表现是否相似？为了进行这一比较，我们引入了两种变体，以高效地将词元对数几率转换为字节对数几率：1）近似方法：Marginalize-It，以及2）精确方法：End-Of-Token。随后，我们首次进行了大规模研究，对仅解码器的密集Transformer模型进行过度训练，同时改变两个维度：词元化方案（词元、字节、带EOT的字节）和训练目标（蒸馏与交叉熵），扫描了约10亿参数、层参数匹配的模型，数据量高达1万亿字节。在跨越三个类别（多项选择问答、语言生成和机器翻译）的八个基准测试中，我们发现Token-1B模型在低FLOPs区域表现优于字节模型（End-Of-Token-1B和Bytes-1B），但最终趋于平稳；字节模型起初表现较差，但随着计算量的增加而超越Token-1B模型，达到更高的下游任务性能上限。将平均top-1误差与验证BPB的扩展规律进行外推，预测渐近情况下，蒸馏的End-Of-Token-1B比蒸馏的Token-1B性能高出最多4%。它们的数据效率也高得多，仅使用六分之一的训练数据即可匹配蒸馏Token-1B的性能。此外，通过在256字节的小词汇表而非约10万词元上操作，它们避免了在对数几率转储期间进行top-k截断的需要，同时将对数几率存储成本降低到约五分之一。最后，我们的下游性能扩展规律预测，蒸馏的End-Of-Token-1B模型在平均下游任务上渐近地超越Llama 3.2-1B、Gemma-3-1B-pt和Gemma 2B模型，分别最多达6.5%、8.1%和2.1%。
英文摘要：
Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.
