arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.12907cs.LGcs.CL
收藏

调和 Kaplan 与 Chinchilla 缩放定律

Reconciling Kaplan and Chinchilla Scaling Laws

Microsoft Research(微软研究院);MIT(麻省理工学院)

Tim Pearce, Jinyeop Song

AI总结:

本文指出 Kaplan 与 Chinchilla 缩放定律差异主要源于 Kaplan 只统计非嵌入参数且在小规模下分析,通过模拟验证后重申 Chinchilla 系数,并建议未来缩放研究使用总参数和总计算量。

摘要:

Kaplan 等人 [2020](“Kaplan”)和 Hoffmann 等人 [2022](“Chinchilla”)研究了在下一词元语言预测任务上训练的 Transformer 的缩放行为。这些研究对在给定计算预算($C$)下,应如何设置参数数量($N$)和训练词元数量($D$)以实现最低可能损失给出了不同估计。Kaplan:$N_\text{optimal} \propto C^{0.73}$,Chinchilla:$N_\text{optimal} \propto C^{0.50}$。本文发现,这一差异的很大一部分可归因于 Kaplan 统计的是非嵌入参数而非总参数,同时其分析是在小规模下进行的。在这些条件下模拟 Chinchilla 研究会产生接近 Kaplan 的有偏缩放系数。因此,本文通过解释 Kaplan 最初高估的主要原因,重申了 Chinchilla 的缩放系数。作为第二项贡献,本文解释了所报告的损失与计算之间关系的差异。这些发现使我们建议未来的缩放研究使用总参数和总计算量。

英文摘要:

Kaplan et al. [2020] (`Kaplan') and Hoffmann et al. [2022] (`Chinchilla') studied the scaling behavior of transformers trained on next-token language prediction. These studies produced different estimates for how the number of parameters ($N$) and training tokens ($D$) should be set to achieve the lowest possible loss for a given compute budget ($C$). Kaplan: $N_\text{optimal} \propto C^{0.73}$, Chinchilla: $N_\text{optimal} \propto C^{0.50}$. This paper finds that much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale. Simulating the Chinchilla study under these conditions produces biased scaling coefficients close to Kaplan's. Hence, this paper reaffirms Chinchilla's scaling coefficients, by explaining the primary cause of Kaplan's original overestimation. As a second contribution, the paper explains differences in the reported relationships between loss and compute. These findings lead us to recommend that future scaling studies use total parameters and compute.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑