arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.19913cs.LGcs.AIcs.CL
收藏

跨Token时间跨度的最优学习率缩放

Scaling Optimal LR Across Token Horizons

Microsoft(微软);Nvidia(英伟达);Meta

Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, Xia Song

AI总结:

本文通过大规模实证研究发现LLM训练中最优学习率随Token时间跨度显著变化并遵循缩放定律,据此可从短跨度准确估计长跨度最优学习率,且提供了零开销的转移经验法则,并指出LLama-1学习率过高及其性能损失。

摘要:

最先进的LLM由缩放驱动——即缩放模型规模、数据集规模和集群规模。对最大规模的运行进行广泛的超参数调优在经济上不可行。相反,近似最优的超参数必须从较小的实验中推断或“转移”而来。Yang等人已研究了跨模型规模的超参数转移。然而,跨数据集规模(或Token时间跨度)的超参数转移尚未被研究。为弥补这一空白,我们进行了一项大规模实证研究,探讨在LLM训练中最优学习率(LR)如何依赖于Token时间跨度。我们首先证明,最优学习率随Token时间跨度显著变化——更长的训练需要更小的学习率。其次,我们证明最优学习率遵循缩放定律,且通过此类缩放定律,可以从较短时间跨度的最优学习率准确估计较长时间跨度的最优学习率。我们还提供了一条在当前实践基础上零开销的跨Token时间跨度转移学习率的经验法则。最后,我们提供证据表明LLama-1使用了过高的学习率,并估计了由此带来的性能损失。因此,我们认为跨数据规模的超参数转移是LLM训练中一个重要但被忽视的组成部分。

英文摘要:

State-of-the-art LLMs are powered by scaling -- scaling model size, dataset size and cluster size. It is economically infeasible to extensively tune hyperparameter for the largest runs. Instead, approximately optimal hyperparameters must be inferred or \textit{transferred} from smaller experiments. Hyperparameter transfer across model sizes has been studied in Yang et al. However, hyperparameter transfer across dataset size -- or token horizon -- has not been studied yet. To remedy this we conduct a large scale empirical study on how optimal learning rate (LR) depends on token horizon in LLM training. We first demonstrate that the optimal LR changes significantly with token horizon -- longer training necessitates smaller LR. Secondly we demonstrate the the optimal LR follows a scaling law, and that the optimal LR for longer horizons can be accurately estimated from shorter horizons via such scaling laws. We also provide a rule-of-thumb for transferring LR across token horizons with zero overhead over current practices. Lastly we provide evidence that LLama-1 used too high LR, and estimate the performance hit from this. We thus argue that hyperparameter transfer across data size is an important and overlooked component of LLM training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑