arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.00448cs.LGcs.CL
收藏

超越Chinchilla最优:在语言模型缩放定律中考虑推理成本

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

Databricks MosaicML

Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle

AI总结:

本研究针对现有LLM缩放定律忽略推理成本的问题,修改Chinchilla定律以优化给定质量和推理需求下的模型参数与训练数据,通过计算预算和实际成本分析及模型训练验证,为高推理需求场景提供更小、训练更长的模型选择依据。

摘要:

大型语言模型(LLM)缩放定律是估计模型质量随参数数量和训练数据增加而变化的经验公式。然而,包括Deepmind Chinchilla缩放定律在内的这些公式忽略了推理成本。我们修改Chinchilla缩放定律,以计算在给定质量和推理需求下训练和部署模型的最优LLM参数数量和预训练数据规模。我们从计算预算和实际成本两方面进行分析,发现预期较大推理需求(约10亿次请求)的LLM研究人员应训练更小、训练时间更长的模型,而非Chinchilla最优模型。此外,我们训练了47个不同大小和参数数量的模型来验证公式,发现当每参数令牌数扩展到极端范围(高达10,000)时,模型质量持续提升。最后,我们对拟合Chinchilla缩放定律系数的过程进行消融实验,发现仅从典型令牌/参数比收集的数据推导缩放定律会高估极端范围下额外令牌的影响。

英文摘要:

Large language model (LLM) scaling laws are empirical formulas that estimate changes in model quality as a result of increasing parameter count and training data. However, these formulas, including the popular Deepmind Chinchilla scaling laws, neglect to include the cost of inference. We modify the Chinchilla scaling laws to calculate the optimal LLM parameter count and pre-training data size to train and deploy a model of a given quality and inference demand. We conduct our analysis both in terms of a compute budget and real-world costs and find that LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal. Furthermore, we train 47 models of varying sizes and parameter counts to validate our formula and find that model quality continues to improve as we scale tokens per parameter to extreme ranges (up to 10,000). Finally, we ablate the procedure used to fit the Chinchilla scaling law coefficients and find that developing scaling laws only from data collected at typical token/parameter ratios overestimates the impact of additional tokens at these extreme ranges.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑