arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.03009cs.LGcs.CL
收藏

专家混合语言模型升级利用(Upcycling)的缩放定律

Scaling Laws for Upcycling Mixture-of-Experts Language Models

Seng Pei Liew, Takuya Kato, Sho Takase

AI总结:

本研究聚焦大语言模型升级为专家混合模型(MoE)的缩放行为,通过实验得出相关经验缩放定律,发现两类训练数据集存在交互项限制大预算下升级效率,给出缩放指导及升级优于从头训练的预算条件。

摘要:

预训练大语言模型(LLM)资源消耗极高,即便使用高端GPU集群,训练时长也常需数月。缓解此类计算需求有两种途径:复用较小模型训练更大模型(升级利用upcycling),以及训练专家混合模型(MoE)这类计算高效的模型。本文研究LLM升级为MoE模型的过程,其缩放行为目前仍缺乏充分探究。通过大量实验,我们确定了描述性能如何依赖数据集规模与模型配置的经验缩放定律。尤其值得注意的是,我们发现虽然缩放这些因素能提升性能,但密集训练数据集与升级训练数据集之间存在一种新的交互项,会在计算预算较大时限制升级利用的效率。基于这些发现,我们为升级利用的缩放提供了指导,并明确了在预算约束下升级利用效果优于从头训练的条件。

英文摘要:

Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such computational demands: reusing smaller models to train larger ones (upcycling), and training computationally efficient models like mixture-of-experts (MoE). In this paper, we study the upcycling of LLMs to MoE models, of which the scaling behavior remains underexplored. Through extensive experiments, we identify empirical scaling laws that describe how performance depends on dataset size and model configuration. Particularly, we show that, while scaling these factors improves performance, there is a novel interaction term between the dense and upcycled training dataset that limits the efficiency of upcycling at large computational budgets. Based on these findings, we provide guidance to scale upcycling, and establish conditions under which upcycling outperforms from-scratch trainings within budget constraints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑