arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01035cs.CL
收藏

充分利用分词器以优化预训练与领域适配

Getting the most out of your tokenizer for pre-training and domain adaptation

University of Edinburgh(爱丁堡大学);Meta AI

Gautier Dagan, Gabriel Synnaeve, Baptiste Rozière

AI总结:

本文研究分词器设计对大型语言模型性能的影响,提出通过专门化分词器并在大规模微调时切换,以提升生成速度和有效上下文大小。

摘要:

分词是现代大型语言模型中研究不足且常被忽视的组成部分。大多数已发表的工作在所有实验中使用单一分词器,通常借用自其他模型,而不进行消融实验或分析以优化分词。此外,在微调基础模型时,分词器通常保持不变。在本文中,我们展示了分词器的大小、预分词正则表达式和训练数据能显著影响模型的生成速度、有效上下文大小、内存使用和下游性能。我们训练了专门的字节对编码代码分词器,并对分词器设计对代码生成任务(如HumanEval和MBPP)中大型语言模型性能的影响进行了广泛的消融实验,提供了分词器超参数选择和在预训练大型语言模型中切换分词器的建议。我们在从零开始训练的模型和从预训练模型微调的模型上进行了实验,验证了它们对广泛用例的适用性。我们发现,当在超过500亿个令牌上进行微调时,我们可以专门化预训练大型语言模型的分词器,从而在生成速度和有效上下文大小方面获得巨大收益。

英文摘要:

Tokenization is an understudied and often neglected component of modern LLMs. Most published works use a single tokenizer for all experiments, often borrowed from another model, without performing ablations or analysis to optimize tokenization. Moreover, the tokenizer is generally kept unchanged when fine-tuning a base model. In this paper, we show that the size, pre-tokenization regular expression, and training data of a tokenizer can significantly impact the model's generation speed, effective context size, memory usage, and downstream performance. We train specialized Byte-Pair Encoding code tokenizers, and conduct extensive ablations on the impact of tokenizer design on the performance of LLMs for code generation tasks such as HumanEval and MBPP, and provide recommendations for tokenizer hyper-parameters selection and switching the tokenizer in a pre-trained LLM. We perform our experiments on models trained from scratch and from pre-trained models, verifying their applicability to a wide range of use-cases. We find that when fine-tuning on more than 50 billion tokens, we can specialize the tokenizer of a pre-trained LLM to obtain large gains in generation speed and effective context size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑