Tokenization is an understudied and often neglected component of modern LLMs. Most published works use a single tokenizer for all experiments, often borrowed from another model, without performing ablations or analysis to optimize tokenization. Moreover, the tokenizer is generally kept unchanged when fine-tuning a base model. In this paper, we show that the size, pre-tokenization regular expression, and training data of a tokenizer can significantly impact the model's generation speed, effective context size, memory usage, and downstream performance. We train specialized Byte-Pair Encoding code tokenizers, and conduct extensive ablations on the impact of tokenizer design on the performance of LLMs for code generation tasks such as HumanEval and MBPP, and provide recommendations for tokenizer hyper-parameters selection and switching the tokenizer in a pre-trained LLM. We perform our experiments on models trained from scratch and from pre-trained models, verifying their applicability to a wide range of use-cases. We find that when fine-tuning on more than 50 billion tokens, we can specialize the tokenizer of a pre-trained LLM to obtain large gains in generation speed and effective context size.
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers
NEC Laboratories Europe GmbH(NEC欧洲实验室有限公司);University of Mannheim(曼海姆大学);Bosch Center for Artificial Intelligence(博世人工智能中心);Karlsruhe University of Applied Sciences(卡尔斯鲁厄应用技术大学)
Domain adaptation of Russian ModernBERT for long legal documents
Independent Researcher(独立研究者);National Research Nuclear University MEPhI(国立核研究大学 MEPhI);Moscow Center for Advanced Studies(莫斯科高级研究中心);University of Waterloo(滑铁卢大学)
Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean
Department of Big Data, Chungbuk National University(Chungbuk National University大数据系);Institute of Digital Research and Innovation, Cambodia Academy of Digital Technology(柬埔寨数字技术研究院);Department of Management Information Systems, Chungbuk National University(Chungbuk National University管理信息系);BigData Labs Co., Ltd.(BigData Labs公司)
CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Wuhan AI Research(武汉人工智能研究院)
TokAlign: Efficient Vocabulary Adaptation via Token Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院,北京);School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(人工智能学院,中国科学院大学,北京)
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization
Dept. of Computer Science and Engg., IIT Kharagpur(印度Kharagpur理工学院计算机科学与工程系);Dept. of Medicine (Biomedical Informatics), Stanford University(斯坦福大学医学院(生物医学信息学))
Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
LUMIA Lab(LUMIA实验室);School of Artificial Intelligence(人工智能学院);Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);SJTU Paris Elite Institute of Technology(上海交通大学巴黎精英技术学院)
Lite Training Strategies for Portuguese-English and English-Portuguese Translation
Institute of Computing, University of Campinas(坎皮纳斯大学计算研究所);School of Electrical and Computer Engineering, University of Campinas(坎皮纳斯大学电气与计算机工程学院);NeuralMind Inteligência Artificial(NeuralMind 人工智能公司);David R. Cheriton School of Computer Science, University of Waterloo(滑铁卢大学 David R. Cheriton 计算机科学学院)