arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.16829cs.CLcs.AIcs.LG
收藏

理解并缓解语言模型中的分词偏差

Understanding and Mitigating Tokenization Bias in Language Models

University of Toronto(多伦多大学);Meta AI

Buu Phan, Marton Havasi, Matthew Muckley, Karen Ullrich

AI总结:

本文揭示最大前缀编码和字节对编码会给语言模型带来无法靠更多训练或数据消除的采样偏差,并提出无需微调即可获得无偏估计的新算法,从而从分词模型中模拟无 token 行为并准确恢复转移概率。

摘要:

最先进的语言模型是自回归的,并在称为 token 的子词单元上运行。具体而言,在将条件字符串传递给语言模型进行下一 token 预测之前,必须将其编码为 token 列表。我们表明,流行的编码方案,如最大前缀编码(MPE)和字节对编码(BPE),会引入一种采样偏差,这种偏差无法通过更多的训练或数据来缓解。为了应对这一普遍问题,针对上述每种编码方案,我们提出了一种新算法,从任何在分词数据上训练的语言模型中获得无偏估计。我们的方法不需要微调模型,并且其复杂度(定义为模型运行次数)在 MPE 情况下随序列长度线性扩展。因此,我们表明可以从分词语言模型中模拟无 token 行为。我们通过马尔可夫链设置实证验证了方法的正确性,在该设置中,与直接将 token 提示给语言模型的传统方法相比,我们的方法准确恢复了转移概率。

英文摘要:

State-of-the-art language models are autoregressive and operate on subword units known as tokens. Specifically, one must encode the conditioning string into a list of tokens before passing to the language models for next-token prediction. We show that popular encoding schemes, such as maximum prefix encoding (MPE) and byte-pair-encoding (BPE), induce a sampling bias that cannot be mitigated with more training or data. To counter this universal problem, for each encoding scheme above, we propose a novel algorithm to obtain unbiased estimates from any language model trained on tokenized data. Our methods do not require finetuning the model, and the complexity, defined as the number of model runs, scales linearly with the sequence length in the case of MPE. As a result, we show that one can simulate token-free behavior from a tokenized language model. We empirically verify the correctness of our method through a Markov-chain setup, where it accurately recovers the transition probabilities, as opposed to the conventional method of directly prompting tokens into the language model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑