arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.04777cs.LGmath.OC
收藏

大语言模型的优化超参数规律

Optimization Hyper-parameter Laws for Large Language Models

Department of Mathematics, National University of Singapore, Singapore(新加坡国立大学数学系);School of Computing, National University of Singapore(新加坡国立大学计算机学院);Department of Mathematics and Institute of Operations Research and Analytics, National University of Singapore, Singapore(新加坡国立大学数学系和运筹分析研究所);Skywork AI, Beijing(北京Skywork AI)

Xingyu Xie, Kuangyu Ding, Shuicheng Yan, Kim-Chuan Toh, Tianwen Wei

AI总结:

本文提出Opt-Laws框架,通过分析SDE收敛和逃逸特性,预测最终训练损失,从而在小规模实验中预选学习率调度方案,提高了超参数选择的准确性。

摘要:

大语言模型推动了显著的AI进步,但其训练过程资源消耗大且对超参数选择高度敏感。尽管扩展定律提供了模型大小和数据需求的指导,但它们在选择动态超参数(如学习率调度)方面存在不足。为此,我们提出优化超参数规律(Opt-Laws),该框架将最终训练损失作为学习率调度、模型大小和数据大小的函数进行预测。基于SDE基于的收敛和逃逸分析,Opt-Laws产生可解释的收敛和逃逸特征,能够预测不同模型规模下的最终训练损失,从而在小规模实验中预选调度方案。实证表明,Opt-Laws在验证配置上实现了94%的Top-2命中率,正确识别了所有五个评估的非家族设置中的最佳性能调度家族,并以F1=0.92检测到训练发散。

英文摘要:

Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they fall short in choosing dynamic hyper-parameters, such as learning-rate (LR) schedules, that evolve during training. To bridge this gap, we present Optimization Hyper-parameter Laws (Opt-Laws), a framework that predicts final training loss as a function of LR schedule, model size, and data size. Grounded in SDE-based convergence and escape analyses, Opt-Laws yield interpretable convergence and escape features that predict final training loss across model scales, enabling schedule pre-selection from small-scale experiments. Empirically, Opt-Laws achieve a 94% Top-2 hit rate for identifying near-optimal schedule candidates on held-out configurations, correctly identify the best-performing schedule family in all five evaluated out-of-family settings, and detect training divergence with F1 = 0.92.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑