Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size. We investigate how vocabulary size impacts LLM scaling laws by training models ranging from 33M to 3B parameters on up to 500B characters with various vocabulary configurations. We propose three complementary approaches for predicting the compute-optimal vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit of the loss function. Our approaches converge on the conclusion that the optimal vocabulary size depends on the compute budget, with larger models requiring larger vocabularies. Most LLMs, however, use insufficient vocabulary sizes. For example, we predict that the optimal vocabulary size of Llama2-70B should have been at least 216K, 7 times larger than its vocabulary of 32K. We validate our predictions empirically by training models with 3B parameters across different FLOPs budgets. Adopting our predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes. By increasing the vocabulary size from the conventional 32K to 43K, we improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21 FLOPs. Our work highlights the importance of jointly considering tokenization and model scaling for efficient pre-training. The code and demo are available at https://github.com/sail-sg/scaling-with-vocab and https://hf.co/spaces/sail/scaling-with-vocab-demo.
MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
East China Normal University（华东师范大学）
尽管预训练语言模型（PLM）的发展显著提高了各种中文自然语言处理（NLP）任务的性能，但这些中文PLM的词汇仍然是Google Chinese BERT \cite{devlin2018bert}提供的基于汉字的词汇。第二，掩蔽语言模型预训练基于单个词汇，这限制了其下游任务性能。在这项工作中，我们首先提出了一种新的方法，\emph{seg\_tok}，形成汉语BERT的词汇，与中文分词（CWS）和子词标记的帮助。然后，我们提出了三个版本的多词汇预训练（MVP），以提高模型的表达能力。实验表明：（a）与基于字符的词汇表相比，\emph{seg\_tok}不仅提高了中文PLM在句子级任务上的性能，而且还提高了效率：（b）MVP提高了PLM的下游性能，特别是提高了\emph{seg\_tok}在序列标注任务上的性能。
CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference
Department of Computer Science, Yale University（耶鲁大学计算机科学系）；College of Engineering, Columbia University（哥伦比亚大学工程学院）；Department of Statistics, University of Wisconsin-Madison（威斯康星大学麦迪逊分校统计学系）
NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies
Institute for Artificial Intelligence, Peking University, Beijing, China（北京大学人工智能研究院）；Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education（教育部高可信软件技术重点实验室）；School of Computer Science, Peking University, Beijing, China（北京大学计算机学院）；State Key Laboratory of Networking（网络与交换技术国家重点实验室）；School of Computer Science, Beijing University of Posts（北京邮电大学计算机学院）
Bridging the Gap between Different Vocabularies for LLM Ensemble
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Wuhan AI Research（武汉人工智能研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院，北京，中国）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院，北京，中国）
DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Models
Department of Computer Science（计算机科学系）；Aalto University（阿alto大学）；Deep Algorithms and Systems Lab（深度算法与系统实验室）；IST Austria（IST奥地利研究院）；University of Bath（巴斯大学）
TokAlign: Efficient Vocabulary Adaptation via Token Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院，北京）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（人工智能学院，中国科学院大学，北京）
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Kyutai；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Institute of Intelligent Systems and Robotics（智能系统与机器人研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Grenoble INP（格勒诺布尔国立理工学院）；LIG（格勒诺布尔信息学实验室）
MedPath: Multi-Domain Cross-Vocabulary Hierarchical Paths for Biomedical Entity Linking
Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, The Netherlands（阿姆斯特丹大学医学信息学系，阿姆斯特丹大学医学中心，荷兰）；Amsterdam Public Health, Methodology, Amsterdam, The Netherlands（阿姆斯特丹公共卫生，方法学，阿姆斯特丹，荷兰）；ILLC, University of Amsterdam, The Netherlands（阿姆斯特丹大学ILLC，荷兰）
