Large language model (LLM) scaling laws are empirical formulas that estimate changes in model quality as a result of increasing parameter count and training data. However, these formulas, including the popular Deepmind Chinchilla scaling laws, neglect to include the cost of inference. We modify the Chinchilla scaling laws to calculate the optimal LLM parameter count and pre-training data size to train and deploy a model of a given quality and inference demand. We conduct our analysis both in terms of a compute budget and real-world costs and find that LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal. Furthermore, we train 47 models of varying sizes and parameter counts to validate our formula and find that model quality continues to improve as we scale tokens per parameter to extreme ranges (up to 10,000). Finally, we ablate the procedure used to fit the Chinchilla scaling law coefficients and find that developing scaling laws only from data collected at typical token/parameter ratios overestimates the impact of additional tokens at these extreme ranges.
Harbin Institute of Technology(哈尔滨工业大学);Chinese Academy of Sciences(中国科学院);Fudan University(复旦大学);Beijing University of Posts and Telecommunications(北京邮电大学)
Effective Frontiers: A Unification of Neural Scaling Laws
Department of Mathematics and Statistics, Xi'an Jiaotong University(西安交通大学数学与统计学系);Department of Artificial Intelligence, Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);University of Chinese Academy of Sciences(中国科学院大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Language models scale reliably with over-training and on downstream tasks
Columbia University(哥伦比亚大学);UT Austin(德克萨斯大学奥斯汀分校);Apple(苹果公司);University of Washington(华盛顿大学);Juelich Supercomputing Center, Research Center Juelich(于利希超级计算中心,于利希研究中心);LAION;Allen Institute for AI(艾伦人工智能研究所);Stanford University(斯坦福大学);Tel Aviv University(特拉维夫大学);TU Munich(慕尼黑工业大学);Contextual AI
Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);Ant Group(蚂蚁集团)