arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14903cs.CLcs.LG
收藏

分词至关重要:分词方式对前沿大语言模型算术能力的影响

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

Gatsby Computational Neuroscience Unit, University College London(伦敦大学学院盖茨比计算神经科学单元);Google DeepMind(谷歌DeepMind)

Aaditya K. Singh, DJ Strouse

AI总结:

本文首次系统研究数字分词方式对大语言模型算术推理的影响,发现从右到左分词显著优于从左到右分词,并揭示了系统性错误模式及模型规模对分词偏置的缓解作用。

摘要:

分词,即将输入文本切分为输入标记的过程,是大语言模型流程中一个常被忽视的环节,却可能成为有用或有害的归纳偏置来源。历史上,大语言模型一直依赖字节对编码,而很少关注特定输入领域。随着大语言模型在推理任务中的使用日益增多,各种针对数字的分词方案被相继采用:LLaMa和PaLM等流行模型选择逐位数字分词,而GPT-3.5和GPT-4则为每个1位、2位和3位数字分别设置独立标记。本文通过算术任务研究这一选择对数值推理的影响。我们考察了GPT-3.5和GPT-4的从左到右与从右到左分词方式,发现从右到左分词(在推理时通过逗号分隔数字来强制实现)能带来显著的性能提升。此外,我们发现模型在使用标准从左到右分词时的错误呈现出刻板的错误模式,这表明模型的计算是系统性的而非近似性的。我们证明模型能够轻松地在不同分词方式之间进行转换,从而使得受思维链启发的策略能够恢复从左到右分词输入上的性能。我们还发现,随着模型规模扩大,不同分词方向之间的性能差距会缩小,这可能表明更大的模型能够更好地克服这种依赖于分词方式的归纳偏置。总之,我们的工作首次系统研究了数字分词选择如何导致模型在算术任务上的性能差异,并附有对错误模式的深入分析。我们希望这项工作能激励实践者在构建通用数值推理模型时,更细致地消融与数字分词相关的各类选择。

英文摘要:

Tokenization, the division of input text into input tokens, is an often overlooked aspect of the large language model (LLM) pipeline and could be the source of useful or harmful inductive biases. Historically, LLMs have relied on byte pair encoding, without care to specific input domains. With the increased use of LLMs for reasoning, various number-specific tokenization schemes have been adopted, with popular models like LLaMa and PaLM opting for single-digit tokenization while GPT-3.5 and GPT-4 have separate tokens for each 1-, 2-, and 3-digit numbers. In this work, we study the effect this choice has on numerical reasoning through the use of arithmetic tasks. We consider left-to-right and right-to-left tokenization for GPT-3.5 and -4, finding that right-to-left tokenization (enforced by comma separating numbers at inference time) leads to largely improved performance. Furthermore, we find that model errors when using standard left-to-right tokenization follow stereotyped error patterns, suggesting that model computations are systematic rather than approximate. We show that the model is able to convert between tokenizations easily, thus allowing chain-of-thought-inspired approaches to recover performance on left-to-right tokenized inputs. We also find the gap between tokenization directions decreases when models are scaled, possibly indicating that larger models are better able to override this tokenization-dependent inductive bias. In summary, our work performs the first study of how number tokenization choices lead to differences in model performance on arithmetic tasks, accompanied by a thorough analysis of error patterns. We hope this work inspires practitioners to more carefully ablate number tokenization-related choices when working towards general models of numerical reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑