更多算力才是你所需要的
More Compute Is What You Need
AI总结:
本文提出一种统一扩展定律,认为 Transformer 模型性能主要取决于总算力而非模型与数据规模的具体分配,并据此预测推理效率应优先小模型大数据集,而在网络数据耗尽后扩大模型规模是唯一提升路径。
摘要:
大语言模型预训练的成本日益高昂,大多数实践者依赖扩展定律来为模型规模和训练 token 分配算力预算,这通常被称为计算最优或 Chinchilla 最优。在本文中,我们提出一种新的扩展定律假设,认为对于基于 Transformer 的模型,模型性能主要取决于所投入的算力总量,而与模型规模和数据集规模的具体分配方式无关。利用这一统一的扩展定律,我们预测:(a) 为了推理效率,训练应优先选择较小的模型规模和较大的训练数据集;(b) 假设可用网络数据集耗尽,扩大模型规模可能是进一步提升模型性能的唯一途径。
英文摘要:
Large language model pre-training has become increasingly expensive, with most practitioners relying on scaling laws to allocate compute budgets for model size and training tokens, commonly referred to as Compute-Optimal or Chinchilla Optimal. In this paper, we hypothesize a new scaling law that suggests model performance depends mostly on the amount of compute spent for transformer-based models, independent of the specific allocation to model size and dataset size. Using this unified scaling law, we predict that (a) for inference efficiency, training should prioritize smaller model sizes and larger training datasets, and (b) assuming the exhaustion of available web datasets, scaling the model size might be the only way to further improve model performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。