arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04706cs.CL
收藏

LiteToken: 从BPE分词器中移除中间合并残余

LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers

Peking University, Beijing, China（北京大学）

Yike Sun, Haotong Yang, Zhouchen Lin, Muhan Zhang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

LiteToken通过移除BPE分词器中的低频残余标记，减少标记碎片化和参数，提升对噪声输入的鲁棒性，同时保持模型性能。

摘要：

分词对于语言模型如何表示和处理文本至关重要，然而广泛使用的BPE分词器的行为却比模型架构和训练方法受到更少的研究。在本文中，我们研究了BPE词汇中的中间合并残余：在合并学习过程中频繁出现的标记，因此保留在最终词汇中，但在分词使用过程中很少被进一步合并且很少被发射。这些低频标记不仅浪费词汇容量，还增加对对抗性或非典型输入的脆弱性。我们对常用分词器进行了系统性的实证分析，介绍了LiteToken，一种用于移除残余标记的简单方法。由于受影响的标记很少被使用，预训练模型通常可以适应修改后的分词器而无需额外微调。实验表明，LiteToken减少了标记碎片化，减少了参数，并提高了对噪声或拼写错误输入的鲁棒性，同时保持了整体性能。

英文摘要：

Tokenization is fundamental to how language models represent and process text, yet the behavior of widely used BPE tokenizers has received far less study than model architectures and training. In this paper, we investigate intermediate merge residues in BPE vocabularies: tokens that are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage. Such low-frequency tokens not only waste vocabulary capacity but also increase vulnerability to adversarial or atypical inputs. We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens. Because the affected tokens are rarely used, pretrained models can often accommodate the modified tokenizer without additional fine-tuning. Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑