大语言模型的优化超参数规律
Optimization Hyper-parameter Laws for Large Language Models
AI总结:
本文提出Opt-Laws框架,通过分析SDE收敛和逃逸特性,预测最终训练损失,从而在小规模实验中预选学习率调度方案,提高了超参数选择的准确性。
摘要:
大语言模型推动了显著的AI进步,但其训练过程资源消耗大且对超参数选择高度敏感。尽管扩展定律提供了模型大小和数据需求的指导,但它们在选择动态超参数(如学习率调度)方面存在不足。为此,我们提出优化超参数规律(Opt-Laws),该框架将最终训练损失作为学习率调度、模型大小和数据大小的函数进行预测。基于SDE基于的收敛和逃逸分析,Opt-Laws产生可解释的收敛和逃逸特征,能够预测不同模型规模下的最终训练损失,从而在小规模实验中预选调度方案。实证表明,Opt-Laws在验证配置上实现了94%的Top-2命中率,正确识别了所有五个评估的非家族设置中的最佳性能调度家族,并以F1=0.92检测到训练发散。
英文摘要:
Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they fall short in choosing dynamic hyper-parameters, such as learning-rate (LR) schedules, that evolve during training. To bridge this gap, we present Optimization Hyper-parameter Laws (Opt-Laws), a framework that predicts final training loss as a function of LR schedule, model size, and data size. Grounded in SDE-based convergence and escape analyses, Opt-Laws yield interpretable convergence and escape features that predict final training loss across model scales, enabling schedule pre-selection from small-scale experiments. Empirically, Opt-Laws achieve a 94% Top-2 hit rate for identifying near-optimal schedule candidates on held-out configurations, correctly identify the best-performing schedule family in all five evaluated out-of-family settings, and detect training divergence with F1 = 0.92.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。